Abstract
Blockchain technology has gained substantial traction in recent years, revolutionizing industries through its decentralized and trustless nature. However, the security of blockchain systems and, specifically, their underlying incentive mechanisms remains a critical concern, potentially leading to catastrophic losses if those economic incentives shaping the behavior of rational participants are not aligned with the expected behavior. This thesis addresses this challenge by proposing a comprehensive framework that ultimately leverages MARL to enhance the security of blockchain incentives while not being strictly limited to it. We propose a generic blockchain model that encapsulates the core components of blockchain systems, making it flexible and easily adaptable to diverse blockchain designs. This model serves as a foundational framework for enhancing blockchain security. Based on this model, we define a taxonomy of incentive vulnerabilities in blockchain systems. This classification categorizes, ranks and prioritizes vulnerabilities based on their feasibility and network impact. The taxonomy aids in identifying critical areas of interest where automatic and potentially exploratory work might be required to assess the resilience of the system. To complete the framework, we introduce a blockchain simulator that is, by construction, as close as possible to the generic blockchain model but, it is also ex-act with respect to specific protocols, and, compatible with reinforcement learning, allowing us to replicate real-world scenarios using honest or byzantine agents with arbitrary objectives. The same simulator can then be used to assess the effectiveness of a given countermeasure. Finally, we show a concrete study of protocol vulnerability of Ethereum 2.0 using all of the above contributions, serving as a practical example of the usage of MARL in the context of the proposed framework. By integrating these contributions, this thesis contributes to the evolving field of blockchain security and provides the means for developers and researchers to analyze, identify and address incentive vulnerabilities in a standardized and systematic manner. Using MARL as a security enhancement tool offers promising results, paving the way for more robust and secure blockchain systems in the future.