AI-Ready Data: The anthology

Excitement around data engineering topics is typically limited to data engineers, and their data engineering-adjacent peers. Terms such as “tagging”, “metadata” and “pipelines” are often used without much explanation, which can create gaps in understanding across technical and non-technical teams. 

That gap can be detrimental to an organization’s technical maturity, and at the very least increases frustration levels across technical and non-technical employees. This tension often bubbles up to funding decisions, when those holding the purse strings are trying to determine which initiatives to sponsor and which ones may have to wait until next time.

Unfortunately, the repeated deferral of putting money into foundational technical efforts can also cause frustration across both technical and non-technical employees. What may be a nightmarish system for a data or platform engineer to maintain often propagates as a slow and stale system to marketing or accounting teams. 

This is part one of a two-part effort to bridge the gap between technical and non-technical team members around the concept of AI-Ready Data (AIRD). As the name suggests, this is all about ensuring the data AI systems rely on is fit for use by those systems. This is a super-hot topic among the Data and AI community (we’re fun), but may not be fully understood by the users and sponsors of AI systems. The anecdotes presented here aim to increase that understanding through short stories. These pieces cover the foundations of AI-ready data – part one focuses on the prerequisites for a data-first organization.

Meet the new boss, same as the old boss

AIRD characteristics are aligned with the trends we’ve observed through the era of the cloud data platform. It just calls on us to do a better job of this.

Step zero to achieving AIRD is of course, data availability. Circa the early 2010s, there was a wave of organizations pushing to get their data migrated to the cloud. Prior to migration, data was in disparate legacy systems; it was difficult for the people working at the company to access this data, let alone to put it to work to actually generate value. This accelerated a broader shift toward cloud data platforms, with organizations seeking easier access to data across the enterprise.

Spoiler: data availability plus a shiny new technology platform was not enough to become a data-first organization. While directionally correct, the same core issues which hindered organizations’ progress in this domain pre-migration did not go away. If anything, they were amplified by the delta between where they were at, and where they could be. So began a ripple effect of follow-on phases to organizations’ data journeys in an effort to better empower their people with their data. History repeats itself, the situation today is similar with the enablement of AI at organizations; however, the pace of progress could be accelerated by proactively implementing the learnings from embracing the cloud.

Data quality for young minds

“Garbage-in, garbage-out” is an overused but nevertheless perfectly true summary of the impacts of data quality on outcomes. Preparing data for AI is like preparing material for early childhood development. While mature adults can consume low-brow content without seriously risking their literacy or articulation abilities, this is not the content you would expose a child to when helping them develop their language and logic skills.

Similarly, the data fed to AI should be curated as much as possible so that the foundational context upon which it is developed is solid. In practice, this means:

• Minimal null values

   

• Consistent null handling (i.e.: ‘’ or NULL/NaN for all nulls, not a combination)

   

• Consistent formats for dates and strings (ie: casing on location names, province/state codes, casing and spacing on zip/postal codes)

The steps required to prepare data for AI must be deterministic: strict, standardized and consistent. Similar to the standardization and consistency embedded in learning curricula for children.

Let them cook: Governance & security

Often, governance and security in AI are immediately linked to ethics or guardrails. While important, these aspects of governance and security cannot be implemented confidently until the underlying data that feeds AI has robust foundational governance and security measures implemented. 

A five-star chef can only guarantee the quality of the meals they prepare based on the quality of the ingredients they sourced. Fine dining restaurants implement rigorous, multi-step processes across sourcing, prep and serving of food. Sanitation and inventory freshness checks occur multiple times per day, every day… Are we still talking about food, or have we moved on to data?

Yes.

Executive and sous chefs may have broader access because their responsibilities span the kitchen; think of these as privileged users. Line chefs and prep cooks only have access to the ingredients required for their line of business; think of these as regular users, as well as AI agents. These kitchens also use locked storage for their most valuable ingredients, impose supplier restrictions such that vendors cannot actually enter the kitchen and only authorized personnel may bring these raw materials in and front of house personnel (who serve patrons) can only deliver the goods once approved by the chef.

In practice, this means:

• Classifying datasets into categories based on risk, value, sensitivity and other relevant criteria, so that the least-privilege access model can be defined against the data

   

• Defining roles and access policies against each type of dataset

   

• Implementing processes such as checks on data freshness, checks on data quality (previously discussed) and checks on data lineage to ensure data is responsibly sourced from upstream suppliers, sanitized by in-house workers and supplied to downstream consumers

Governance and security on the foundational data enables AI to do its job responsibly, so you can let it cook.

Mickey Mouse metadata & cataloguing

Imagine walking into a store in search of one particular item. None of the aisles are labeled and there is no clear grouping of products. Searching for your product feels like a needle in a needlestack. 

Fortunately, there are uniformed employees walking around this store and so you go to ask one of them, whose nametag identifies them as Micky, for guidance. You tell Micky that you’re looking for Product A, but instead of pointing you in the right direction, Micky says, “I don’t know if we sell that here. I recognize we’re the type of store you’d come to for this, but I only manage Product V; try talking to Minnie.” and so Micky directs you to Minnie, with whom you have a similar interaction. 

Then you get passed to Goofy. You’re just about ready to leave when you’re passed back to Micky; when Goofy explains what you’re looking for, Micky exclaims, “Oh! Why didn’t you say that the first time? Product A is actually a subset of Product V, let me show you where it is”.

As you’re walking through the store, you realize that a lot of the products look outdated; in some cases, moldy. And there are no barcodes on anything. You ask Micky about this, and Micky explains that they don’t have a procurement or tracking system for their store. Therefore, whenever someone has the time to do so, they may go clean up a section of the store but it is a highly manual and inefficient effort; besides, most of their time is taken up finding products for customers, so cleanup never really happens.

The lack of barcodes leads you to ask if theft is also an issue. Micky explains that their security guards at the store entrances do a really good job and have caught a few culprits, but they really have no way of knowing for sure.

This store without any inventory management is a data platform without metadata and cataloguing. Without effective metadata and cataloguing, serving data to AI responsibly and at scale becomes significantly more difficult, and extracting reliable value from organizational data becomes much harder. Even if the right steps are taken, the time to market will likely be so long that the potential value of the work is significantly diminished.

The right tools for the job: Semantic business layer 

Most databases, especially long-lived enterprise databases, contain a lot of unclear table names, ambiguous column labels and department-specific logic. Like any newbie to the database (or more likely, databases), this can confuse a large language model (LLM). The result can be incorrect interpretations, calculations or assumptions.

Diana is the new apprentice at a machine shop. She’s bright and has shown lots of promise at her other jobs, but this shop is different, massive and multi-generational.

The master machinist, Clark, calls for a “5/8th bolt”. Clark is very good at his job but can be impatient. Diana looks around the shop, and locates multiple bins full of bolts. 

Bin A is labelled “BLT_58_HEX”; Bin B is labelled “58_THREAD_M”; Bin C looks like it contains identical bolts, but they are actually 16mm fasteners. Beyond this, the plumbers call a pipe fitting a “coupling”, which the hydraulic workers call a “union” and the structural experts call it an “adapter”.

If Diana were to simply guess and grab the bolt from Bin C, she might be able to thread in half an inch before stripping an entire engine block. Diana’s issue was not that she lacked enthusiasm nor intelligence; the problem was that there was no standardized system of specifications. Had the shop placed a standardized spec sheet on the wall with standardized labels and clear guardrails, there would have existed a single source of truth, understandable by all employees.

Likewise, in a database, this situation is also problematic. It takes the shape of duplicate or near-duplicate data which is difficult for machines to distinguish. This kind of ambiguity can increase the likelihood of incorrect interpretations and unreliable outputs.

That’s all for now, folks!

Part One has covered the structural necessities (cleanliness, governance, organization) which create the foundation for an AI platform. Just as a five-star dining experience cannot exist without a clean kitchen, AI systems are much harder to build and operate reliably when the data they depend on is ambiguous or untrusted. 

In Part Two, the anecdotes build on top of these prerequisites by focusing on the operationalization of data to fuel AI applications.