Introduction
Tyler Cowen and Sonia Farrell Pearson have an excellent essay out on Pearson’s Substack, and you should most certainly read it.
It is called “Capitalizing Untethered AI Agents”. Here’s an explanation of what the essay is trying to do, and why, therefore, you should be reading it:
The essay asks a simple question: when, in the near future, AI agents do something that we humans will consider to be a crime, whom do we hold responsible? How do we identify the perpetrator, how do we punish the perpetrator, and how do we think about restitution for the victim?
Now, we live in a world where answering this question requires us to ask some truly fundamental questions. For example: what is AI, and what is an AI agent? Who bears responsibility for the crimes committed by an AI agent? When AI (and therefore AI agents) get increasingly powerful, will we retain the ability to even understand how they have committed crimes, let alone police them or prevent them? Assuming we are able to identify the perpetrator, what will count as meaningful punishment, both in the sense that it should “fit the crime” and act as meaningful, ongoing deterrence? Will our systems have the ability to enforce said punishment?
Tyler and Sonia’s essay addresses some of these questions, and as I said, you should read it to get a sense of their answers to these questions. In what follows, I want to talk about the questions and the objections that I had while reading the essay. There are no easy answers to any of these questions, and I do not mean for these to be “gotcha” questions - in fact, in quite a few cases, these questions have been raised by the authors themselves, and even (partially) addressed by them. But I will still raise them, because they belong to a theme that (Claude-ism alert) forms the spine of this essay, the one that you are reading now.
My Primary Objection
First, the central point, as I understand it, of the essay we are critiquing right now:
The future is likely to bring an enormous number of AI agents. Even if the vast majority are aligned and collaborative, a very small minority of bad or careless actors could wreak havoc. It is, we must remember, much easier to destroy value than to create it.
We thus turn our interest to the concept of capitalization and whether we ought to require that untethered AI agents hold assets. If we could make bad behavior cost agents something, might this be another way of inducing agents to behave? Perhaps. After all, people’s behavior is curbed by fear of consequence, whether social, financial, or legal.
My point is that getting this to happen - that is, having the ability to require that untethered agents hold assets - will require us to have AI agents be available in the institutions that will operationalize this. I hold this to be self-evident; you may or may not. In case you do not, please read Cowen and Pearson’s essay. It contains a pretty good set of arguments for why this is going to be inevitable. And internalizing this point brings us to my fundamental objection:
“What happens when the institution that is supposed to control agents is itself populated by agents?”
But I am getting ahead of myself. First, an excerpt from their essay about this:
In both of these cases, we’ve established that for capitalization to work, AI systems must be involved in the oversight of autonomous agents. There’s some question here about how autonomous these “legal” agents ought to be; in any case, we need to trust they are aligned enough to do the job well, to humanity’s benefit. A successful capitalization regime, then, necessarily presupposes alignment from both the majority of the agent population and its AI enforcers. Turtles all the way down is okay if we can find a reason to trust the turtle we stuck at the bottom more than the one now standing on its back.
In case you are confused by what you just read, here’s the simplified version, in a numbered list:
Tyler and Sonia are worried about AI agents being let loose in our world
In order to “tether” them, let us “capitalize” them. This is one of their main proposals.
But setting up such a system, and even more so, regulating and enforcing it is only going to be possible with the help of agents
But that takes us back to pt. 1, where we should now be worried about the AI agents that we are counting on to help us tether the AI agents that have been let loose in our world.
Those AI agents that we are now worried about in pt. 4? That’s the turtle stuck at the bottom. Do we have a reason to trust it?
Or, if you prefer my framing at the start of this section:
“What happens when the institution that is supposed to control agents is itself populated by agents?”
Consider the section in their essay when they talk about insurance, for example. Which insurance firm goes up against AI agents? Does said insurance firm also use AI agents? And if yes, how does that work? When does the AI agent that is acting as the insurer honor a claim? What about *its* identity - does it persist? Does the AI agent acting as the insurer value money? Under what circumstances will it collude with the insured AI agent? Can these circumstances, and these methods of collusion be guessed at ex-ante? Can they be detected after the fact?
The rest of my essay takes questions along similar lines, and explores their nuances. The nuances truly matter in this case, because my argument rests upon the implication of the governing institution itself being populated at least in part by AI agents. This is going to have an impact upon enforcement by these institutions, and also on money (and therefore on insurance), compliance and jurisdiction choice.
Two Leviathans
At its heart, any legal system must have the ability to identify the fact that a crime has been committed. It must have the ability to investigate said crime. It must then be able to adjudicate in a way that is seen as being consistent, and with rules that are broadly accepted as being legitimate by both parties. Finally, it should have the ability to enforce its judgement.
If I’m going over the speed limit, I have done something wrong. This must be identified, investigated, adjudicated, and a fine must be levied. The point I am making in this section is a simple one: when I’m asked to pay that fine, I don’t blow a raspberry in the face of the law and say “You say I have to pay this fine? You and what army?”
Hobbes answered that question. And his answer has remained the best one in a practical sense. The state’s ability to be the “coercer of last resort” has remained (mostly) unchallenged. To the extent that you accept the notion that anarchy and state power are on opposite ends of the spectrum, you see where I am coming from. Not meaningfully challenged, that is, until now.
What if an AI agent asks “You, and what army?” What if that AI agent’s challenge can only be answered by another AI agent? Why can it be answered only by another AI agent, you ask? Because an AI agent may be able to act in such a way (and at such a rate) that it can think, coordinate, replicate *and* execute its plans faster than any human institution can govern it. That is why the institution governing AI agents will need to be populated, at least in part, by AI agents.
What happens when the institution that is supposed to control agents is itself populated by agents?
This point, to me, is more important than being just one section in one essay which is itself a response to another essay. This is the central challenge of an AI-first world. One sovereign may go up against another, of course, and one nation-state may go up against another. But there has been no challenge to the philosophical concept of Leviathan itself... until now.
There are now two Leviathans in town. Or, if you prefer, there is now a serious, entirely non-human challenge to Leviathan in town. And our planning has to be done with this simple fact in mind.
Resources Matter, But Does That Imply Money Matters?
Capitalizing agents will work if said capital has significant opportunity costs for AI agents. If these agents end up losing this capital (say, because they are fined), that loss should pinch them meaningfully, in that it should deter them from harmful conduct. Money should be a dominant (by which I mean major) way for AI agents to access scarce resources.
So what will AIs be able to buy with money? More importantly, which of these things will AIs want, and how can we think about this in advance? The bad news is that we just don’t know, and that is true of the authors as well. Across two separate footnotes (12 and 13) they acknowledge that the evidence is limited and that we don’t know how (and whether) future agents will value money. As they mention in their essay, building a regime of this sort is not going to be easy, and it is not obvious that we will succeed in building a world in which AIs value capitalization as much as we humans do today. Assuming that we are able to build such a world, capitalizing AIs has its advantages - true. But it remains a very big if.
The central question for this section is this: can an agent obtain what it wants without using monetary assets that can be confiscated?
If the answer to this question is in the affirmative, well, capitalization as a strategy becomes rather weaker. If the answer to this question is no, this essay becomes very, very important. I think it is entirely correct to say that AI agents will value compute, hardware, access to data, energy, and perhaps some other things. I don’t think it is necessarily true that money will be the only, or even the best way, to gain access to these things when it comes to AI agents.
The Nation-State Stack Is Not Enough
I’m loath to talk about this, for two main reasons, although there are many more. First, I’m skeptical that there is a workable, top-down solution to the problem I am about to raise. Second, we have become immune to folks talking about “global” problems.
But this is that rare case where in spite of these issues, it makes sense to raise the specter of this being a “global” problem. The authors talk about a legal system that must be capable of adjudicating consistently and coherently, and this is easy to agree with. The question is, however, about the geographical extent and reach of this system. It is not just that legal systems will have to become better to deal with these challenges, but the differing nature, status, and rates of adaptation of these legal systems across nation-states will create its own set of incentives for these AI agents. And the rate at which these AI agents will be able to “arbitrage” across these different legal systems will be far faster than the rate at which these legal systems can adapt. Unless, of course, you have the legal systems themselves be AI systems, in which case see the section titled “Two Leviathans”.
But the point I am trying to make in this section is this: AI is a planetary, human-scale phenomenon. Its impacts, consequences, effects and second-order effects are going to be felt by every person alive on this planet, regardless of income, nationhood, gender, location and religion. Thinking about dealing with AI, and thinking about living with AI by using the tools made available to us by the nation-state stack is not going to be enough. By the nation-state stack, I mean a system in which sovereign states remain the highest units that make, enforce, and negotiate the rules. That doesn’t necessarily imply the need for a world government, but it does very much imply a minimum global layer of coordination.
What is optimal at the nation-state level may not be optimal, and may in fact be counter-productive at the global level, and vice-versa. Expecting global coordination to be seamless, rapid and fluid has been a non-starter at the best of times; these qualities are table stakes in the age of AI. Something’s gotta give, and it isn’t going to be AI. In passing, note that one interpretation of increasing signs of global coordination is that frontier AI models are causing deep disquiet at the highest levels of sovereign state governments.
By the way, I do not mean to imply that jurisdictional competition is necessarily bad. Experimentation is all to the good, and having different institutional models compete can only lead to good outcomes. But Coase, alas. A country may capture the benefits of hosting agents while exporting part of the resulting risk to everybody else. Given the speed and scale at which this can and will happen, that minimum global layer of coordination becomes important.
Always Look On The Bright Side of Life
This is perhaps a relatively minor quibble, perhaps a rather more important point, but consider this paragraph from their essay:
“What we need is not a metaphysically individuated agent, but a gated one that needs (or at least prefers) to exist within a single legal structure. So the goal becomes making it as difficult as possible to operate without one.”
I would modify that a little bit. The goal isn’t just about making it as difficult as possible to operate without one, but also about making it as easy as possible to operate within one. That is, agents shouldn’t just think about the costs of not complying, but should also be thinking about the benefits of complying. Moreover, we should be thinking as hard about providing the benefits of being compliant as we are about framing the costs in the case of non-compliance.
The standard framework to use is a simple one:
The benefit of being inside (less the cost of compliance) must be more than the benefits of being outside (less the costs of evasion). This implies that we have to figure out ways to:
make membership more valuable;
make compliance cheaper;
make evasion more expensive;
reduce the value of the outside option.
My Conclusion About Their Conclusion
Here are the last two paragraphs of their essay:
The one thing we are certain about is that these agents will get more capable. The rest, with varying degrees of confidence, we’ve assumed: that agents will want money, that most agents will be prosocial, and that, for prosocial agents, money will have a restraining effect. Examined, these assumptions ring of us: this is what we expect of people. If these assumptions hold – most remain reachable by the system, most are prosocial, most are constrained by threats to their capital – then capitalization could be a promising path towards inducing better behavior. It is by no means a perfect solution, but we are not going to get very far if perfect is the standard.
But we do not know just how like us they are. We are, in the end, betting that we can apply the logic of human institutions to AI. It is a great deal to assume about minds we do not yet understand, and yet the plan cannot be to wait until we do.
In times such as these, when the age of AI is upon us, it makes sense to examine every single one of our unstated assumptions. For example, consider the assumption that the nation-state stack is optimal for dealing with AI. How realistic is it? How realistic is it to expect there to be a global coalition instead? Most worryingly, how bad do things have to get before folks realize that the nation-state stack isn’t enough? Should we, in such a world, be updating our priors about nation-states, about AI, or both?
More generally, if we find out that the AIs are not like us, do we need to worry about AIs being deviant, or do we need to worry about our institutions not being good enough? What does good mean in the age of AI, and is our answer to this question a function of the times we live in?
But more narrowly still, in response to the authors’ point about capitalization, my take is a very Tylerian one. *At the margin*, capitalization is likely to make agents who are mostly cooperative to begin with safer. Similarly, the proposed strategy also has the potential to provide assets which can be used to compensate victims.
But I do think the issue of the turtle at the bottom remains a rather pressing one, and is not adequately addressed in this essay. And addressing it opens a whole new can of worms, but that is the subject of many other essays.
Our institutions are going to become agentic. That has to be a working assumption. As Roon is fond of saying, whatever level you happen to be thinking at in the age of AI, go one level up. Neither this capital essay by Cowen and Pearson nor my analysis of it is exempt from that requirement.