Is AI governance even the right conversation?

We may be making a category error. Not a small one, either; a foundational one. The debates unfolding around AI — who governs it, which institutions respond, how regulation catches up, whose values get encoded — are real debates. They’re also debates that were calibrated on an older kind of problem, where the challenge, however large, reshapes the world around it but doesn’t reshape itself: where the thing being governed is unchanged while humans struggle to catch up with its effects but where the window for change remains open.

This week offered several signals that this particular frame might not actually fit. They arrived from different directions, including financial markets, enterprise operations, a frontier lab co-founder, and even the Vatican They kept circling the same possibility: that the category of problem may have changed, and the inherited response infrastructure may be insufficient.

The financial picture of our current AI moment

Let’s start with the financial picture, as it’s ultimately the least sentimental one. The S&P 500's Shiller CAPE ratio crossed 40 this week — a level reached only once before in 140 years of data, in 1999. What the number reveals isn't primarily crash risk, but instead the structure of the bet: capital is pricing AI transformation as if the economic benefits are large, fast, and broadly distributed. This is all while OpenAI — the company most directly positioned to capture that value — lost $1.22 for every dollar it earned in Q1 2026. Ramp's AI Index shows enterprise adoption crossing 50% this month, while Uber has already blown its 2026 AI budget and the fastest-growing vendors on enterprise spend platforms are inference providers offering cheap open-source model access. 

The pattern that emerges isn’t conclusive, but the signal is clear: yes, deployment is happening, but the cost isn’t what was planned and the gap between what capital is pricing and what organizations are actually experiencing may be wider than the bet assumes. The economics of the transformation are being negotiated in real time — and they don’t seem to be settling where the market expects.

The market gap and AI elitism

That gap isn’t incidental. It’s the economic expression of a deeper condition: the decisions about how AI gets deployed, at what pace, into which systems, with what constraints, are being made by the smallest possible group of people, inside incentive structures that cannot fully assess the consequences of those decisions for everyone else. 

This is a structural observation — one made, this week, by one of the people inside those structures. Chris Olah, co-founder of Anthropic, stood beside Pope Leo XIV at the Vatican and said that every frontier AI lab operates inside incentives that can conflict with doing the right thing — and that it is enormously important that there be people outside those incentives, willing to say hard things, willing to be earnest critics. 

Whether or not it is a marketing moment, it’s consistent with a structural failure the people inside these institutions are finding it increasingly difficult to ignore: the people making the decisions know the decisions shouldn't be theirs alone, yet they are making them anyway, because the alternative is someone else making them faster. Indeed, Dario Amodei said on CBS last November that he is deeply uncomfortable with these decisions being made by a few companies and, indeed, a few people.

The Vatican’s contribution to the AI debate

The Vatican's participation in this conversation is easy to dismiss as symbolic, but it shouldn't be. Pope Leo XIV published Magnifica Humanitas on May 25, but was signed on the 15th,  the 135th anniversary of Rerum Novarum, the document his namesake wrote in response to industrialization. The date may not have been a coincidence; the suffering caused by industrialization preceded the moral framework — child labor, unsafe conditions, the unmediated exploitation of labor — was not incidental to the timeline. It was the condition that made the framework politically possible. The choice of date may suggest the current Pope is saying that sequence cannot run again. The window for the moral framework to arrive before the damage accumulates isn’t 35 years; it may not even be 35 months.

That framing is urgent. But it still assumes the sequence is available — that a moral framework, however accelerated, can arrive and apply. It still assumes the window stays open.

Here’s where the historical analogy, which is instructive up to a point, begins to break down. Humanity has absorbed civilizational shocks before. Industrialization, nuclear weapons, mass media — the sequence was consistent. Deployment, suffering second, framework, institutional response (eventually). It always arrived late; people died in the gap, but it arrived. The reason it could arrive is that the thing being governed was external to the governing. Locomotives didn’t design better locomotives. Nuclear weapons didn’t improve their own yields. The capability was powerful or grew linearly at a sustained pace. Human institutions, however slowly, could pursue it.

The recursive capability is different in kind, not just degree. A system that applies its own intelligence to improving that intelligence does not wait for the moral framework to catch up. It continues while the framework is being debated, while the institutions are forming, while the philosophers are being hired. The window doesn’t just narrow, it’s closed. It is a logical description of what several frontier labs are actively building — and what Andrej Karpathy joined Anthropic specifically to accelerate: a team that uses Claude to make Claude better. That’s not an ambition statement. It’s a description of how fast the capability is moving — independently of whether the governance conversation has caught up. And yet the conversation continues as if the window remains fully open — as if the thing being governed is still waiting patiently for the governance to arrive.

From debate to institutional action 

The debates currently happening — in policy forums, academia, corporations and even Vatican encyclicals — are asking how to govern this. Who should be in the room. Which values should be encoded. How to make institutions move faster. These are real questions. But they share an assumption that is worth examining: that the thing being built remains, in some meaningful sense, a tool. Powerful, fast, consequential — but a tool. Something humans deploy toward ends they have already defined.

What's not being asked is what happens if that assumption is wrong. Not in the catastrophic science fiction sense, but in the plainest philosophical one: we may be building a system capable of exceeding human intelligence across every domain simultaneously. Not eventually, but possibly within a generation.

And we have not seriously asked what that means for humanity's purpose alongside a system that can do everything we can do, better? How do we coexist with it? Who serves whom? These are not apocalyptic framings. They are the oldest questions philosophy has, arriving now with stakes they never had before.

The governance debate — faster institutions, better frameworks, more voices in the room — is a real and necessary debate. But it is happening entirely beneath that question, as if the question doesn't exist yet. As if we can decide how to govern the deployment before deciding what we are deploying toward. The concentrated power, the institutional inertia, the competing ideologies — these are the permanent conditions of human coordination, and they will shape whatever response emerges, as they always have. But they are not the reason the response may be insufficient. The reason is simpler and harder: we are not yet asking what this is actually for, what we want it to be, and what we are willing to become alongside it.

Technical alignment is a necessary condition. It is not a sufficient one. Alignment to what, decided by whom, toward which definition of what humanity is and wants — that is the question the current conversation keeps approaching and stepping back from. Not because it is unanswerable. Because it is uncomfortable in a way that governance frameworks, deployment standards, and capability disclosures allow us to defer.

The Pope chose that date because the last time a framework arrived 35 years late, people died in the gap. But even that framing assumes the gap is still the kind we have survived before — one where the damage is absorbed and the species continues. The harder possibility, the one this week kept circling without landing on, is that we are in genuinely new territory: not because the technology is more powerful, but because for the first time the thing we are building could redefine the terms of the question itself. And we are still conducting the debate as if it cannot.

That question is not yet being asked seriously enough to produce an answer. And it needs to be named plainly: we may be building a form of intelligence capable of outperforming us in every domain, and we have not yet seriously asked what that means for humanity — not how to govern it, but what we want to become alongside it, and whether we have thought carefully enough about why we are building it at all. The governance debate is real and necessary. But it may be a conversation about the frame while the more important conversation — the one about what we are actually doing and what it means for our species — is still waiting to begin