Businesses have more data, and data capabilities, at their disposal than ever. Yet many are still struggling to bridge the disconnect between awareness of the value of data in theory, and realizing that value in practice. Whether it’s to support strategic decision-making or participate in emerging industry ecosystems, the ability to rapidly access and experiment with a critical mass of relevant and trustworthy data is crucial - and that requires reinvention.
The data disconnect
i. Data lakes everywhere (and not a drop to drink)
A legacy bias towards centralization in the form of data lakes or warehouses is preventing organizations from developing effective approaches to data they need at scale, and from disseminating strategically important information to the teams who need it the most. Consolidation also leads to the creation of architectures and organizational structures that are resistant to change, hampering the enterprise’s ability to test and react to market developments.
“We have to challenge this very fundamental assumption that for any company or business unit to engage in data-driven experimentation they must have access to centralized data to get any meaning out of it."
Director of Emerging Technologies, Thoughtworks North America
ii. Shifting towards a mesh
To unlock the value of data, enterprises need to break out of the monolith, and make data available to the teams who actually know and use it. Based on the principles of domain-oriented data distribution and treating data as a product that delights its consumers, the mesh is a new approach to data architecture, purpose-built to support a resilient, fast-acting, digital business.
iii. Technology foundations
The world of data storage and computation has changed, and companies are no longer necessarily subject to old constraints. Adopting a more distributed data architecture may not be as risky, expensive or time-consuming a proposition as business leaders think. Leading enterprises have built on existing cloud technologies to develop sophisticated platforms that contribute directly to their strategic goals.
Data platform model
iv. Strategy and people
Because teams have typically been structured around legacy data resources, moving to Data Mesh inevitably requires organizational change, as well as careful thinking about the connections between data and business strategy. Business leaders may encounter resistance as they redraw the rules on how data is controlled and shared, and how the performance of some teams is measured - but the end is almost certain to justify the means.
"The truth is, people look at data and particularly data governance and their general reaction is to groan. These perceptions have to change, so people are interested, want to use and consume the data, and are excited about the possibilities."
Head of Technology, Thoughtworks India
v. Security and governance
In any initiative that touches on data, research shows security and risk management will almost certainly be among executives’ chief concerns. Contrary to perceptions, taking responsibility for data out of the hands of a single, centralized control tower and ‘federating’ ownership among domain teams creates positive security and governance impacts, by assigning control of the data to those who know it best, and fostering a broader focus on data accuracy and privacy.
Top data security concerns of CIOs/IT leaders
vi. Dealing with the data talent shortage
While fretting about data skills remaining in short supply, many companies have yet to recognize the resources already at their disposal. Platforms and models are evolving in a way that’s making them more user-friendly, and opening the possibility of training non-specialists in the basics of fields like data engineering and data science. Data Mesh supports this trend by emphasising accessibility and consumer experience - and, sometimes, allowing people to break things.
Biggest technology skills shortages for businesses
vii. Brace for brave new approaches
The acceleration of data demands will continue to present challenges for businesses, but developments in architecture are just one reason for optimism about what comes next. Computing and hardware are evolving at an incredible rate, and there are pools of insight-rich data that remain largely untapped. In this race, fearless thinking and a willingness to strike partnerships will help businesses claim the rewards.
“We need to stop thinking of other businesses as living in different worlds and start to see them as potential partners, finding ways to augment each other with data. Recognizing those benefits requires bold thinkers who are willing to do challenging and complicated things and make that investment."
Principal Data Scientist and Head of Data, Thoughtworks Germany
Perspectives delivered to your inbox
Timely business and industry insights for digital leaders.
The Perspectives subscription brings you our experts’ best podcasts, articles, videos and events to expand upon our popular Perspectives publication.