A working definition of data modernization, and how to tell when it's actually time for your org to pursue it.
The core architecture choices (warehouse, lake, lakehouse) and how a cloud migration usually plays out when it's done safely.
The real hurdles teams hit when modernizing legacy data and code, and more importantly, how teams get past each one.
What "AI-ready" data actually means, and what has to be true before agents can safely act on your data.
A short list of things you can act on this week, not just think about.
Data modernization means improving how an organization collects, manages and uses data so it can meet changing business needs. It covers quality, access, governance, scalability and speed, along with the systems and practices that support them.
Moving data to a newer platform may be part of that work. But if people still cannot find it, trust it or use it to make decisions, the address has changed. The problem hasn’t.
The starting point is a business need: faster reporting, lower operating costs or reliable data for a new service. That need should determine what changes and how success is measured.
Example: Migrating a legacy transactional database (built for fast, small reads and writes) to something like Databricks for analytical workloads. The two jobs, running the business transaction by transaction and analyzing years of it at once, need fundamentally different engines. Trying to make one system do both usually means neither job gets done well.
When the current setup can no longer uphold the business's needs: reports take too long, decisions wait on data that isn't there yet or the system simply can't do what the business now needs it to do.
When the cost of running the current setup (maintenance, licensing, the specialists needed to keep it alive) is meaningfully higher than a modern alternative would cost.
Business growth: when the organization can see that its future needs will look different from its current ones, and waiting until the old system breaks is more expensive than moving ahead of it.
Worth knowing: when 350 senior data and technology executives were surveyed on why they modernize, AI wasn't the top answer. Better decision-making across the business was (cited by 46%), ahead of supporting AI models (40%). AI is a real driver, but if AI is the only reason on your list, you're probably underselling the rest of the business case.
Data warehouse: structured, schema-on-write, built for business intelligence (BI) and reporting with mature SQL tooling. A good fit when your workloads are well-defined, mostly structured and governance/consistency matter more than raw flexibility.
Data lake: cheap storage, schema-on-read, holds structured, semi-structured and unstructured data side by side. Great for data science and exploratory work, but without discipline, it turns into a data swamp nobody trusts.
Lakehouse: the industry's answer to not wanting to run and reconcile two separate stacks. It layers warehouse-style governance and transactional guarantees on top of lake-style storage and flexibility. Most modern platforms (Databricks, Snowflake, Fabric, BigQuery) have converged here, usually organized using a medallion architecture: raw data lands in a bronze layer, gets cleaned into silver and is aggregated into business-ready gold tables.
The honest answer to "which one should we pick" is: look at your workload shape first. Structured, BI-heavy and compliance-sensitive tends to lean warehouse. Exploratory, ML-heavy and format-varied tends to lean lake. Most orgs today land on a lakehouse specifically to avoid maintaining both.
Cloud migrations usually get triggered by one of a few things:
The cost of running on-prem infrastructure.
The need for elastic compute, especially for AI/ML training, which is expensive to run on-prem.
A vendor sunsetting a legacy platform.
Simply not being able to hire people who still know the old stack.
Whatever the trigger, resist the urge to migrate everything at once and flip a switch. Trying to build one complete, cleansed, enterprise-wide data platform before delivering a single use case is one of the most common ways these programs stall. The cleanup always takes longer than expected, the business loses patience and budget gets redirected before value ever shows up.
A phased migration (one workload or domain at a time, with validation against the legacy system and a rollback path if reconciliation checks fail) is slower on paper and considerably safer in practice. It's also how you keep the option to change course without losing everything you've already built.
(This is exactly what the validation-and-rollback loop in the architecture diagram above is doing. It's not an afterthought; it's the safety net that makes phased migration possible.)
A migration can finish on schedule and still produce data nobody trusts. These four hurdles determine whether the new system preserves business logic, produces reliable results and remains maintainable after launch.
The problem: incomplete documentation, missing data flow diagrams, the people who understood these legacy data systems already gone, and a team that now needs both the missing context and migration-specific skills nobody currently has.
How teams get past it: capture what departing experts know before they're gone. It's cheaper than reverse-engineering it later. Use AI-assisted code analysis to reconstruct data lineage and surface business rules buried in old ETL logic, paired with human subject-matter expert (SME) review, not blind trust. Blend internal domain knowledge with external migration specialists, and let AI agents (with human oversight, not a rubber stamp) handle the mechanical code conversion, freeing your scarce engineers to validate business logic instead of retyping it.
The problem: maintainability quietly takes a back seat during migration. Bugs surface, refactoring becomes the next task and fixing one thing correctly means fixing something else first. It becomes a cycle.
How teams get past it: set a non-negotiable test coverage bar before code counts as "migrated": running is not the same as correct. Add automated regression tests that compare old-system output against new-system output for the same inputs, and enforce lint, style and complexity gates in CI so debt doesn’t quietly reaccumulate in the new stack the same way it did in the old one.
The problem: data can pass technical checks and still be wrong for the business. A populated field is not necessarily an accurate one.
How teams get past it: agree with business users on what makes a dataset fit for its intended use, including accuracy, completeness and freshness. Build those expectations into automated checks, with clear owners and a response when a check fails. Give teams closest to the data responsibility for its quality, supported by shared standards. A dashboard full of green ticks means little if the business still disputes the numbers.
The problem: without reconciliation, there's no way to trust that a migration didn't silently break something, and broken trust in the data is expensive to earn back.
How teams get past it: build automated, scalable source-to-target reconciliation into the pipeline itself, not a one-time spreadsheet exercise the week before go-live. Pair it with a rollback mechanism so a failed check halts promotion into production instead of quietly shipping bad data downstream. Teams that get this right don't just migrate faster. They earn trust from day one, because users can see the new system was actually checked against the old one, not just declared done.
The stakes here are real: in a 2025 survey of 1,000 senior decision-makers, 42% reported negative ROI from their data modernization efforts. The gap usually isn't a technology gap. It's a strategy and execution gap, and the hurdles above are exactly where that gap opens up.
Even a small migration error can disrupt reporting, operations or revenue. Good planning matters, but so does limiting how much can go wrong at once. Choose an initial scope small enough to validate and recover, yet meaningful enough to prove business value.
Identify a domain or subdomain that’s clearly outgrown its current setup (the one causing the most pain, or carrying the most business value) and start there. Before you touch anything, define what "done" and "rollback" look like for that phase. Ship it, validate it against the old system, bank the win and only then move to the next domain.
This is the same logic behind delivering in thin slices rather than front-loading years of infrastructure work before anyone sees value: small, provable wins build the internal trust needed to fund the rest of the roadmap. It's a lot easier to ask the business for the next domain's budget when you can point to a working one, than to ask for a blank check up front for a multi-year rebuild.
Governance needs to work when data is accessed or an action is taken. Give people and agents only the permissions their tasks require. Set shared policies across the organization, with domain teams responsible for applying them to their data.
Automate access checks and policy enforcement where possible, and log activity so teams can investigate failures or misuse. Periodic reviews still matter: permissions drift, business needs change and controls need testing. A policy document cannot stop an unauthorized action. An enforced access rule can.
An agent needs more than access to accurate data. It needs enough context to interpret that data and clear limits on what it can do with it. A revenue figure is useful only if the agent knows which period it covers, how it was calculated and whether it is approved for the task.
Context engineering means supplying the information and tools an agent needs for a specific task. That can include business definitions, metadata, source references and access to governed datasets. Tools exposed through the Model Context Protocol (MCP) can help connect agents to those sources, but the connection itself does not make the data trustworthy.
Validate inputs, check outputs against task requirements and enforce permissions when tools are used. For consequential actions, define when human approval is required and how errors will be detected and contained.
Good data is necessary, but it does not guarantee sound judgment. The question is not just whether an agent can find an answer. It is whether that answer is reliable enough for the action that follows.
The distinction between connecting tools and controlling their use is consistent with MCP’s own guidance on human oversight.
Once data is quality-checked, governed, reconciled and discoverable, here's the actual payoff:
Analytics and BI that people believe. Trustworthy dashboards, built on clean customer data, mean the finance, sales and ops teams stop showing three different 'revenue' numbers in the same meeting.
AI and machine learning (ML) models that are worth the compute spent training them. Models amplify whatever they're fed. Bad data compounds faster through a model than it ever did through a static report.
Agentic use cases that are actually safe to run. Agents acting semi-autonomously across systems is only a reasonable thing to allow once the governance and quality work above is genuinely in place. This is a prerequisite chain, not a shortcut you can skip to.
Operational efficiencies that show up in hours, not as a line item that keeps getting pushed to the next meeting. Less time spent reconciling conflicting numbers in meetings, less duplicated data lingering across teams, faster time from question to decision.
Build quality, governance and discoverability into each use case as you deliver it. You do not need to finish modernizing the entire estate before improving a report or testing an AI application. You do need to establish that the data is fit for the task before relying on the result. Otherwise, you are simply automating the confusion.
Defining, capturing and, more importantly, actually following a source of truth while building data assets pays off in three concrete ways:
Trustworthiness: users of a data asset know exactly what kind of information and data type to expect from it.
Discoverability: data assets can be found by name, description or the columns they provide, instead of by asking around.
Reusability: if an existing data asset already meets a business need, there's no reason to spin up a near-duplicate one, which only adds data duplication and, eventually, confuses whoever (human or AI agent) has to pick between them later.
Two open standards, ODCS (Open Data Contract Standard) and ODPS (Open Data Product Standard), are how this stops being a principle and becomes something enforceable. Write the ODCS contract before you build the pipeline, not after: schema, quality rules and SLA go in first, the same way two services agree on an API contract before either one is built. That contract then becomes the actual thing your validation and reconciliation checks test against, instead of a rule someone remembers to apply manually. Because it's plain YAML, it lives in version control next to the code, and a breaking schema change fails a pull request instead of failing silently in production three weeks later.
ODPS works a level up. Instead of documenting a data product in a wiki page nobody keeps current, you define it once: ownership, ports, support channels and semantic context. That file becomes what both people and AI agents actually query to find and trust the asset. It's the discoverability problem mentioned above, solved with a machine-readable file instead of tribal knowledge.
Both specs:
Data modernization isn’t about chasing the newest stack. It’s a business-aligned journey through real legacy hurdles (codebase, skills, code quality, data quality, reconciliation) toward data that’s quality-checked, governed and ready for both people and AI agents to use with confidence. The teams that get the most value out of it are the ones who move in phases, validate as they go and treat "we can roll this back" as a feature, not a fallback.
Three things you can act on this week:
Pick one domain or subdomain that's clearly in pain today, and scope a single phased migration for it.
Capture what your legacy code and data flows actually do, right now. Use AI-assisted analysis to accelerate it.
Sit down with the business and define what "valid" looks like for your single most-used dataset and put some automated checks on it.
Choose based on your workloads, governance needs, team skills and operating costs. Warehouses often suit structured reporting, lakes offer flexibility across data formats and lakehouses combine elements of both. Test representative workloads before committing.
The timeline depends on the scope, legacy dependencies and regulatory requirements. An estate-wide program may take years, so define smaller milestones that deliver measurable improvements along the way. Set expectations for the first useful outcome as well as the full program.
No. Better decisions, reliable reporting, compliance and cost control can each justify modernization. AI adds another reason to improve data foundations, but the business case should identify the outcomes that matter to your organization.
It means data is fit for a defined AI use case: accurate and current enough for the task, clearly described, discoverable and accessible under appropriate permissions. For agents, readiness also depends on controls over how that data informs decisions and actions.
Two open standards from the bitol-io project, the Open Data Contract Standard and the Open Data Product Standard, for describing data assets and the contracts around them in a consistent, machine-readable way. Early-stage, but relevant if you're treating data as a product.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
