Abstract
Resistance to chemotherapies, particularly to anticancer treatments, is an increasing medical concern. Among the many mechanisms at work in cancers, one of the most important is the selection of tumor cells expressing resistance genes or phenotypes. Motivated by the theory of mutation-selection in adaptive evolution, we propose a model based on a continuous variable that represents the expression level of a resistance gene (or genes, yielding a phenotype) influencing in healthy and tumor cells birth/death rates, effects of chemotherapies (both cytotoxic and cytostatic) and mutations. We extend previous work by demonstrating how qualitatively different actions of chemotherapeutic and cytostatic treatments may induce different levels of resistance. The mathematical interest of our study is in the formalism of constrained Hamilton-Jacobi equations in the framework of viscosity solutions. We derive the long-term temporal dynamics of the fittest traits in the regime of small mutations. In the context of adaptive cancer management, we also analyse whether an optimal drug level is better than the maximal tolerated dose.