The Data Mesh Shift

What is Data Mesh?

A domain-driven data architecture that is treated as a product and owned by teams that most intimately know and consume the data.

Today, data is ubiquitous. Data is the by-product of any and every action we take. Everything, every system, every process, every sensor generates data. Technology makes it easier for organizations to collect and store data, for businesses to leverage to make better decisions or create more tailored experiences for their customers.

However, organizations are struggling to enable and empower their employees to make the most informed and timely decisions possible. Centralized data platform architectures fail to deliver data with the speed and flexibility scaling organizations need.

Data mesh is a response to this problem. Data mesh applies the principles of modern software engineering and the learnings from building robust, internet-scale solutions to unlock the true potential of enterprise data.

Domain ownership

Several companies have reorganized into product teams for business domains, but still have a centralized data platform owned by a centralized data team. The Data Mesh approach allows product teams to own the data they produce — in a controlled and compliant way. This approach brings together data knowledge to speed up the delivery of value with data. How can the components of the data ecosystem and their ownership be decentralized? What capabilities might be needed to enable domain teams to create and manage their data products?

Data as a product

A shift to product thinking for data enables better data quality by creating accountability with the data product owner. Apply domain-driven design to identify data products — which should be cohesive and independent in the domain context. What is a “data product”, exactly? What does it mean to be a data product owner? Who can be a data product owner? How do data product owners collaborate?

Self-serve data platform

Access to data typically requires multiple conversations, tickets and approvals. Data mesh seeks to eliminate the friction to deliver quality data for producers and enable consumers to discover, understand and use the data at rapid speed. This approach ensures compliance and security and can significantly lower the lead time to get insights from data. What does it take to make data sets independently understandable? How do you ensure appropriate access controls over the data? How would you manage the cost of infra on-prem or on the cloud?

Federated computational governance

Traditional data platforms tend to default to centralized data governance. Data Mesh requires a Federated Computational Governance model that maintains global controls while increasing local adaptability. Semantic standards, security policies and compliance are managed centrally while the responsibility to comply is decentralized to data product owners. How would data governance units be run? How are data product owners organized to ensure accountability? How are security policies enforced?