Abstract
Standard protein substitution models use a single amino-acid replacement rate matrix which summarizes the biological, chemical and physical properties of amino acids. However, site evolution is highly heterogeneous and depends on many factors: genetic code, solvent exposure, secondary and tertiary structure, protein function, etc. These impact the substitution pattern, and in most cases a single replacement matrix is not enough to represent all the complexity of the evolutionary processes. This paper explores in a maximum-likelihood framework phylogenetic mixture models, which combine several amino-acid replacement matrices to better fit protein evolution. We learn these mixture models from a large alignment database extracted from HSSP, and test the performance using independent alignments from TreeBase. We compare unsupervised learning approaches, where the site categories are unknown, to supervised ones, where in estimations we use the known category of each site, based on its exposure or its secondary structure. All our models are combined with gamma distributed rates across sites. Results show that highly significant likelihood gains are obtained when using mixture models, compared to the best available single replacement matrices. Mixtures of matrices also improve over mixtures of profiles in the manner of the CAT model. The unsupervised approach tends to be better than the supervised one, but it appears difficult to implement and highly sensitive to the starting values of the parameters, meaning that the supervised approach is still of interest for initialization and model comparison. Using an unsupervised model involving 3 matrices, the average AIC gain per site with TreeBase test alignments is 0.31, 0.49 and 0.61, compared to LG, WAG and JTT, respectively. This 3-matrix model is significantly better than LG for 34 alignments (among 57), and significantly worse for 1 alignment only. Moreover, tree topologies inferred with our mixture models frequently differ from those obtained with single matrices, indicating that using these mixtures impacts not only the likelihood value but also the output tree. All our models and a PhyML implementation are available from http://atgc.lirmm.fr/mixtures.