Abstract
In the modern world data is considered to be a new, highly valuable asset that underpins the decision-making and innovation process. At the first glance, it is valuable for companies because it provides a way to evaluate the internal business processes, to find inefficiencies, and to improve the delivered services and products. Meanwhile, it is also valuable for research institutions that collect and analyze the data about the experiments or the world around, and thus enabling us to prove or to discard the theoretical proposals based on the practical validations and facts. Data can be represented in many forms: articles, messages, measurements, music, wave signals, etc; but generally it is characterized by having the large volume, different degree of veracity, changing creation velocity, variety of its forms, and ultimately hidden value that modern organizations try to extract from it.In this thesis, we focus on the development and enhancement of the decentralized data mesh platform that incorporates a data-as-a-product philosophy. More concretely, our efforts are dedicated to the theoretical formalization and practical implementation of the federated computational governance aspect of the data mesh which is still considered to be an unresolved issue as of today. To achieve that, we shift our attention away from the widely used data management techniques like centralized data catalogs or role-based authorization and look at the field that natively supports the decentralization mechanisms - we study the blockchain networks.Indeed, the application of the distributed blockchain ledger as a backbone for metadata management and policy enforcement looks attractive, but also very challenging. For these reasons, the first contribution of this thesis lays down the theoretical generalization of multiple data platform governance modes and the justification of using the ledger and smart contracts as core governance mechanisms. Furthermore, it outlines the operation and the integration of the advanced, cloud-native data orchestration framework with our blockchain-based metadata system which enables organizational scaling, data security and ownership. Consequently, we provide a proof-of- concept system and several real-world user scenarios that are used as a validation to our novel proposal.In the context of our second contribution, we attempt to improve the data discovery and product navigation process of the aforementioned federated metadata system by virtue of transforming the ledger records to the labelled property graph model. Since the graph databases are commonly used as a data cataloguing solution, it seems quite natural to maintain its benefits in regard to natural modeling, storage, and processing of the cross data product relationships. Hence, our main efforts are focused on retaining the advantages of both the blockchain and the graph data structures. As part of it, we propose the rules and algorithms that fulfill three required properties of the database mapping, that are: computability, semantic and information preservation. Additionally, we develop an implementation of the concurrent algorithm for generic and schema-dependent mapping and provide the benchmarking evaluation.The last part of this thesis is dedicated to the description of the global, end-to-end data mesh platform that is built upon our contributions. By succeeding in integration of the multiple systems, namely the federated blockchain-based governance, the customizable and extensible data processing platform, and user friendly data discovery interface, we demonstrate a novel solution that advances the current research in data platform architectures. The thesis concludes with achieved objectives, discusses the issues and challenges that are still faced by the scientific community, and proposes a future vision and planned actions on the way to resolving these questions.