AI and ML
Using Grafana AI to accelerate incident investigation
Delivery speed and AI ethics dominate the conversation around AI adoption. Far less attention goes to a critical question: who is accountable when an AI agent fails in production? Increasingly, that responsibility falls to product leadership, making AI risk a product decision, not just a legal one.
For years, regulatory risk sat with legal and compliance. With the EU AI Act and GDPR now shaping what products can do, that has changed. The choices that determine your exposure are made in product discovery, architecture and vendor selection, long before legal sees them.
That puts Chief Product Officers (CPOs) in a new position. The aim isn't a defensive moat that slows delivery. Teams that can show clear oversight, controlled data use and traceable decisions earn enterprise trust faster, and that is a commercial advantage as well as a compliance one.
Under the AI Act, fines reach €35m or 7% of global turnover for prohibited practices, and up to €15m or 3% for most other obligations. Where an AI system also processes personal data, one incident can fall under both the EU AI Act and GDPR, enforced by different authorities, so assess exposure under each. The exposure is real, but it's manageable if you design for it.
Both GDPR and the EU AI Act define specific roles and assign different legal responsibilities to each. Just as GDPR distinguishes between data controllers and processors, the EU AI Act distinguishes between providers and deployers. However, product teams risk blurring these boundaries and inadvertently assuming responsibilities associated with a different role.
Two questions every CPO should be able to answer
1. What role do we hold, and what risk class are we in?
Your obligations depend on two things: your role (provider or deployer) and the risk category of each AI feature.
A company that uses a third-party tool internally is typically a deployer, with lighter obligations. A company that builds AI into its own product and ships it under its own name is generally a provider, even when the underlying model comes from a vendor. You can also move from one role to the other: substantially modifying a system, or repurposing it for a high-risk use, can shift provider responsibilities onto you.
What to ask your teams
For each AI feature, which role do we hold, which risk tier is it in and what would change that?
2. How much autonomy are we giving our agents?
Generative AI that drafts or summarizes is one risk profile. Agents that execute multi-step actions across systems are another. Agents are happy-path thinkers; this makes them brittle. They won't flag security boundaries, consent requirements or compliance constraints unless they are engineered to.
Under GDPR, the organization remains accountable as controller for personal data processed by its systems, including by agents acting on its behalf. A failure that exposes personal data brings exposure under Article 32 (security) and Articles 33–34 (breach notification). Many organizations invest heavily in training people to handle personal data, but are only just beginning to recognize the magnitude of the time required to give agents an equivalent level of scrutiny.
What to ask your teams
What can each agent do without a human approving it? What credentials does it hold, for how long? Can we show exactly what it did and why?
Product architecture considerations
To meet these legal demands while reassuring enterprise buyers, CPOs and Heads of Product need to embed governance mechanisms directly into core product specifications:
Narrow non-human identity (NHI) scope
Issue short-lived, task-scoped credentials for every sub-agent and tool invocation, and mandate human-in-the-loop (HITL) approval for high-impact or irreversible actions. Avoid allowing agents to inherit broad, standing user OAuth tokens.
Build audit schema
Internal model "chain-of-thought" outputs or LLM reasoning traces are not a reliable audit record. Product specifications must require explicit authorization schemas that pair user consent context, unique agent identity and the exact tool payload executed.
Isolate context boundaries
Architecturally, context windows treat system prompts, RAG data, and user inputs as a single flat space. Implement prompt-shaping gateways to sanitize sensitive data before payloads leave the trust boundary, and enforce strict tenant-ID filtering across all vector indexes and agent caches.
Treat artifacts derived from AI in the same way as you would personal data
This relates directly to GDPR exposure. Deleting personal records from a relational database does not fulfill GDPR Article 17 ("Right to Erasure") if derived vector embeddings, agent memory states or model weights may retain that data.
Maintain a versioned data bill of materials to track data lineage across the full AI lifecycle.
What the board will need to hear
CEOs need confidence that CPOs have properly assessed and accounted for these risks. This requires three commitments:
Regulatory clarity: For every AI feature, CPOs must be able to demonstrate their regulatory role, its exposure rating and who made the call. Using a vendor model unmodified keeps our obligations lighter, but we remain the provider of any feature we ship under our name.
Operational control: The AI drafts, analyzes and recommends but human-in-the-loop oversight remains required for high-stakes actions, ensuring we avoid handing the legal authority to an autonomous algorithm.
Customer trust and competitive advantage: Roughly 85% of consumers will not do business with companies if they doubt their data practices, especially in heavily regulated sectors like healthcare and finance. While competitors stall due to regulatory confusion, our verifiable data safety, clear oversight and transparency turn compliance into an active sales accelerator.
What this means for product managers
You won't control engineering architecture, procurement policy or legal interpretation. But product managers sit at the intersection of all three, making them the most important point of ownership.
Historically, PMs have trusted tech leads with architectural decisions. The balance is shifting here with PMs getting more involved in these decisions. Architectural choices influence commercial risk, which puts them squarely in the PM's territory. Your job as CPO is to equip PMs to engage early, before architecture is locked in. It isn't helpful identifying legal and ethical risk after the product has been delivered.
Operationally speaking, there are three things PMs need to do:
Nudge your tech leads during architectural discussions: PMs must bridge business risk and technical implementation by asking targeted questions during early system design and architectural spikes:
"Are we fine-tuning our consumption of APIs in a way that shifts us into 'Provider' status?"
"Do our agents use broad standing user tokens, or can we scope them to short-lived credentials with explicit human-in-the-loop checkpoints?"
"How does the cache architecture handle the right to be forgotten across derived content, not just relational databases?"
NFRs that are shaped by compliance: Product managers must translate regulatory constraints into explicit engineering NFRs, acceptance criteria and definition-of-done (DoD) checklists within Jira or Notion. This ensures that privacy-by-design and security boundaries are baked in from day one rather than bolted on as expensive fixes after release.
Templates for vendor procurement: Ensure that your procurement guidelines and third-party evaluation templates evaluate AI vendor SDKs (software development kits), vector stores and model APIs against corporate risk profiles prior to contract signature.
Feature assessment checklist
As we introduce agents and supplier agents into the mix, this space becomes more complex. How do you assess the level of risk when you don't know what features are using agents and which are not? To assist product owners to navigate this complex space, we've developed the tool below.
When considering a feature always consider the four roles:
Controller: We decide why and how personal data is used.
Processor: We handle data for a customer, on their instructions.
Provider: We build the AI and release it under our name.
Deployer: We use someone else’s AI.
If your feature... | We become... | Design lever to lower it | Risk |
Uses customer data only to deliver the service | Processor | Keep it that way: no secondary use | Low |
Uses customer data for our own goals (analytics, improving the product, training models) | Controller for that use | Make it opt-in, anonymise data, or leave it out | High |
Decides what data is kept, for how long or who else receives it | Controller | Let the customer set retention and recipients | Medium |
Handles sensitive data (health, biometrics, children) or tracks people at scale | Controller, with heavier duties: need a privacy impact assessment (DPIA) before build | Collect less; avoid sensitive data unless essential | High |
Makes automated decisions that significantly affect people | Same role, plus people can ask for human review | Recommendation, don’t decide; add human approval | High |
Lets an AI agent choose its own data, recipients or goals | We stay accountable; a processor will risk inadvertently becoming a controller | Limit agent’s permissions and scope; log its actions | Medium-High |
Uses someone else’s AI as it is | Deployer | Follow the vendor’s instructions; keep records of what's been done | Low |
Shows AI output or generates content for users | Same role, plus tell people content is AI generated | Label AI output clearly | Medium |
Uses AI in hiring, credit, education or essential services | High-risk AI: human oversight, logs, informing people | Keep a human decision-maker; scope the feature away from these uses | Medium-High |
Builds, fine-tunes, rebrands or repurposes AI under our name | Provider (we take on the maker’s duties) | Use the vendor’s system unmodified, under their name | High |
Does something banned (social scoring, emotion detection at work or school, manipulation) | Cannot ship | Redesign | Critical |
How this would work in practice: An illustrative example
The head of customer success at a software company suggested an AI assistant that would personalize its answers using each customer’s support history.
The initial product requirements document (PRD) proposed fine-tuning a model on support interactions from all customers to improve answer quality. Checked against the feature assessment, this raised three problems.
Training on customer data served the company’s own goals, which made it a controller for that use.
Fine-tuning a vendor model risked making it the provider under the AI Act.
And once personal data has been used to train a model, deleting the source records may not satisfy an Article 17 erasure request.
The product team considered an alternative: using the vendor model as it is, and storing each customer’s data in a separate RAG index that the model consults when answering, rather than learns from. Customer data would only be used to deliver the service, and an erasure request could be applied to the index as well as the source records.
Under the organization’s internal risk assessment framework, the change reduced the feature’s risk rating from high to medium. As a result, executive sign-off was no longer required, and the feature could proceed through routine review.
This illustrates how a single architectural decision can reduce privacy risks, simplify internal approvals and help product teams deliver faster.
* This article contains opinion and perspective, not legal advice.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
