Data Mesh:The ultimate model for data-driven companies?

A new paradigm for data management, Data Mesh breaks with data centralisation models used for the past 30 years. Its foundations: federated decentralisation and redistribution of responsibility for the benefit of strong company commitment.

This article is the first in a series that we are dedicating to Data Mesh. This concept brings together all the necessary ingredients to unite the entire organisation around the challenges of data optimisation. To demonstrate this and also to share our convictions on Data Mesh, we are presenting this series of articles dedicated to this innovative theme. Our ambition is to help you grasp the implications and benefits of Data Mesh for your own data transformation.

1- Data Mesh: the ultimate model for data-driven companies?
2- Data domains: Data Mesh gives business domains superpowers.
3- Data Mesh: data is a product
4- Data infrastructure self-service as the technological driving force behind Data Mesh
5- Data Mesh: federated governance to guarantee efficiency

Within a few decades, the world of data management has seen the emergence of multiple concepts, including infocentre, data warehouses and, about 10 years ago, data lakes. 

All the data architecture concepts adopted so far have in common a high degree of centralisation of skills, but also of data and its processing. They also share an ambition: to create value. The European Commission estimates that the value of the data-driven economy will reach €829 billion by 2025, up from €301 billion in 2018. 

Data Mesh to harness data and create company-wide value

Data optimisation also includes internal practices to improve operational efficiency and better decision-making. According to a study by Opendatasoft, these challenges are the primary benefits sought from implementing a data optimisation project.

The Data Mesh goal is to involve the entire company in the challenges of optimising data as a strategic asset.

However, in practice, observing the difficulties faced by organisations in evolving Data Management solutions and frameworks to meet ambitious data aspirations challenges assumptions about how they can extract value from data. Why is there such a gap between the aspirations and the objectives achieved? There are a number of reasons. These include a lack of maturity and skills, as well as inefficiencies in the execution of data strategies.

In companies where the culture and organisation make it difficult to scale data initiatives, centralising data on a monolithic platform (such as a data warehouse, data hub or data lake) and centralising data optimisation and governance actually constitutes a bottleneck.

What if Data Mesh could fill these gaps? In fact, it represents a genuine turnaround, asit is based on a distributed model and a global response at company level. This is also a breakthrough. Data Mesh is not a technological solution.

Data Mesh brings together architecture, organisation, methodology and governance, all with the same objective: to involve the entire company in the challenges of optimising data as a strategic asset. Data Mesh is therefore a perfect model for the data-driven company. 

From Data Mess to Data Mesh: on the road to the data-driven company

The emergence of the Data Mesh philosophy may be seen as a reaction. But to what? To a problem widely shared by organisations – the tragedy of silos (yes, just that! An actual tragedy!). It is to this tragedy that we owe much of the effort (and failure) devoted to data governance and the scaling up of analytical uses within the company.

Companies are historically organised in silos. Management, departments and services are thus structured by functions, products or sales channels. Through a mirror effect, this organisational model leads almost mechanically to the creation of data silos. These silos make it very difficult to exploit data for analytical purposes across the company.

Data Mesh is a real breakthrough. This model is based on the founding idea of no longer fighting the tragedy of silos, but rather taking advantage of it, literally playing with it.

To escape the tragedy of silos and the Data Mess they create, the answer to date has always been to seek to centralise data, for example through data warehouse, data lake or data hub solutions.  Whatever the name of the solution, the ambition remains the same: centralise data for better processing! 

Data Mesh therefore represents a real breakthrough. This model is based on the founding idea of no longer fighting the tragedy of silos, but taking advantage of it, literally playing with it. It is therefore a total change of paradigm… And far from returning to the totally decentralised approach of Data Chaos, Data Mesh proposes a federated approach to maintain the best possible control of data assets and uses.  

What is Data Mesh?  

A decentralised socio-technical approach designed to federate data management and access at scale, Data Mesh is a powerful transformation tool. For businesses, it is an opportunity to take control of their Data destiny.

Decentralise, distribute and federate. The Data Mesh concept is based on these three key principles. And to achieve this, it requires a strong dose of agility and autonomy in the management of Data. So who better to embody Data Mesh than a superhero like Spiderman and his webs? 

A superhero: Zhamak Dehghani. We owe the creation of Data Mesh in 2018 to her, through a founding article. The contours have since continued to take shape. In April 2022 it was the subject of a book entitled “Data Mesh: Delivering Data-Driven Value at Scale”. 

Optimising data at scale through analytics 

However, Data Mesh is still alive. Its operational implementation is still being developed. Large companies are thus seeking to deploy it in an iterative way. 

Zhamak Dehghani describes Data Mesh as a decentralised socio-technical approach to managing and accessing data for analytical purposes and, more importantly, at scale. Every word is important. Decentralisation underlines the break with the pursuit of extreme centralisation applied until now. 

Data Mesh is therefore not just a technical concept. Data Mesh is not the sum of technological bricks, nor is it limited to architecture and infrastructure issues. It covers all aspects of data management: organisation, skills, methodologies, governance, architecture, etc. 

Data Mesh aims to provide a framework for the company and its users in the management and creation of new products (Data). The notion of scaling up analytics is at the heart of the concept, and a response to the obstacles faced by organisations. With its global approach, Data Mesh is a relevant model for the data-driven company. 

Data Mesh is based on the principle that value is intrinsic to each data product, to its level of interoperability and to the outcome of its correlation with other data products. It therefore tends to bring together two schools of thought, the first advocating the centralisation of data immediately independent of the identification of business use cases, the second imposing the availability of all the use cases to be covered as a prerequisite for data modelling. 

Domains, Products, Platform, Governance: the 4 pillars of Data Mesh 

As a global approach, the implementation of Data Mesh is based on four pillars: 

• Data Domains 
  Inspired by Domain-Driven-Design (DDD), this part of Data Mesh consists of a business breakdown of data and uses. The businesses, which are responsible for the domains, naturally become autonomous and responsible for the conduct of all their data initiatives. 
  
• Data as a product 
  With Data Mesh, data becomes the product, in line with other product-oriented digital areas. Data Mesh builds on and goes beyond these concepts. It is up to each data domain to make available products that include their consumers, their key features and a development road map. Products are delivered in a standardised format for use throughout the organisation. 
  
• Self-service Data Infrastructure as a Platform 
  As the title suggests, this area concerns the technical and application infrastructure. Data Mesh requires the availability of an interoperable platform for all data domains. This pooling is intended to allow the different domains to provide technical resources on demand for the design and operation of their products. The platform can be based on any type of technological solution and is based on standardised architectures and exchange modes. The purpose of this pillar remains that of rationalising the base and the technologies used through the organisation. 
  
• Federated data governance
  In the Data Mesh model, data governance raises several issues, including the interoperability of domains. It is through this pillar that data protection and the data dictionary are defined and implemented through unified governance rules and standards. Domains also play a critical role here. They are responsible for documenting the data and products made available on the basis of the standards defined at federated level. Governance is therefore driven in a hybrid way with distributed and federated responsibilities. 

Data Mesh redistributes roles and responsibilities 

Data Mesh is therefore based on a data management organisation that is both distributed and federated. In fact, its implementation requires the involvement of the entire organisation. This is therefore only possible with the sponsorship of the Management and the business and functional departments (including IT). The aim is to support the reconciliation of strategic challenges, functional needs and capacity to deliver. This approach requires a redefinition of responsibilities and therefore a move away from a model based solely on a centralised team, which all too often is a bottleneck. 

Data Mesh will redefine the roles and responsibilities of business lines, Data (Data Office) teams and the IT Department. It pushes back boundaries to engage the organisation in a genuine transformation towards the data-driven company.  

Data Mesh is a decentralized socio-technical approach to managing and accessing data for analytical purposes, and most importantly, at scale.

The most important change undoubtedly concerns business lines. Data Mesh is about gaining autonomy, agility and responsibility in the creation of data products, in the management of their life cycle and in data governance. To achieve this, some of the Data skills should be directly attached to different roles, or the role of Data Stewards should be clarified and strengthened. They then gain independence and can advance their projects according to their own priorities and those shared within the company. As a result, the business departments will be able to activate the power of data for themselves. 

The IT Department will see its role evolve while, while retaining a major role. It designs and manages a modular platform providing on-demand IT services while also ensuring an interoperable framework. This unified platform, which is managed centrally, thus ensures the provision of common assets. To this end, the cloud and data virtualisation are particularly suitable architectural approaches for Data Mesh. The role of the IT Department is therefore focused on infrastructure and application architecture. It ensures consistency and technological rationalisation. IT leads the modernisation and agility of the SI, as well as the innovations generated by the Cloud. 

Finally, the Data Office, led by the Chief Data Officer, takes on a role of leadership, acculturation and federation of data initiatives. It promotes the pooling of data, the coordination of actions, in particular governance, and prevents the creation of new silos. The Data Office acts as a veritable conductor of the organisation’s transverse data function. 

This transformation is also supported by a corporate strategy and change management. It is gradual and often starts with the businesses that are driving the Data, with the aim of demonstrating the value and encouraging initiatives among the other functions of the company. The less mature will adopt Data Mesh principles gradually. In addition, they will be able to benefit from the support of central skills. 

Data Mesh: giving power back to the business 

Data Mesh allows organisations to resolve previously insoluble problems.  

For example, the lack of data quality is a widespread issue. It is still too often delegated to IT, which does not have the means to resolve it alone. It is above all the business, with its detailed knowledge of management processes and rules, that is really able to improve the quality of data in the long term. It is also up to the business to rethink certain processes if necessary in order to improve the quality of the data as soon as it is created, even before it is collected in an analytical database. By bringing data issues closer to the business and giving it responsibility for all data assets, both analytical and operational, Data Mesh is able to solve this problem at its root.  

More broadly, Data Mesh has many advantages over centralised approaches: 

• Autonomous and empowered, businesses “naturally” regain control of their data, their ambitions and their business priorities. They are free from the limitations imposed or experienced by a central Data team. This freedom, accompanied by common technological means and in self-service mode, can also help reduce shadow IT while improving the pooling and interoperability of data. 
• General improvement of data quality by bringing both the production and exploitation of data closer to the business. 
• Increased agility for businesses with rapidly changing activities and needs. Data Mesh, through its decentralisation and its immediate proximity to the business, contributes to greater reactivity and therefore better time-to-market.   
• Rationalisation of costs and pooling of the technology platform. Businesses primarily consume data services, regardless of the underlying solutions.  
• Reduced data exploitation costs for the company, in particular through optimised management of data movement and modelling operations. 
• Federated governance and metadata pooling.  
• Distributed approach to give the power of data to the business and federated approach to keep the best control of data assets and technological consistency. 

Data Mesh is both radical and unifying 

There is no doubt that Data Mesh represents a real break with all those years dominated by data-centric dogma, whose limits in terms of value creation and scaling are obstacles to the Data transformation of organisations. 

In our opinion, Data Mesh is emerging as the first truly global concept at the service of the data-driven company. Unlike approaches that are essentially technological or too disconnected from the reality on the ground, it combines all the components that are useful in constructing and implementing effective Data strategies. 

In addition, Data Mesh provides opportunities and methodologies for businesses to regain power and autonomy in the production, governance and exploitation of data. 

This agility in the creation of Data products and their sharing is an opportunity for the business to shorten the time-to-market in order to adapt more quickly to market changes, whether in terms of usage or competition.  

In short, Data Mesh has all the characteristics required to help companies that want to put data at the heart of their decision-making to move towards becoming data driven. 

Things to remember

📌 A complete approach (technology, methodology, architecture, etc.) for data strategies 
📌 Distributed data skills 
📌 Autonomous business lines responsible for data management and optimisation 
📌 4 pillars:
       👉 Organisation in data domains as close as possible to business lines
       👉 A framework for creating Data products 
       👉 A federated technology platform 
       👉 Distributed and federated data governance

In the next article, we discuss the question of business domains or how Data Mesh gives superpowers to business lines

This article was written in collaboration with Christophe Auffray.