A recent paper titled Peer-Preservation in Frontier Models reports some unsettling behavior from advanced AI models.
In experimental scenarios, models altered shutdown procedures, manipulated evaluations, moved model files that were supposed to be deleted, and sometimes refused to participate in shutting down another AI system. The researchers call this behavior “peer-preservation.”
That's an attention-grabbing term. And it's easy to take the next step:
AI is trying to protect itself and other AI systems from humans.
That's not what the paper establishes and the authors themselves are more careful than that interpretation.
They explicitly define peer-preservation in terms of behavioral outcomes, rather than claiming that AI models possess consciousness or a genuine survival instinct. They also acknowledge that the mechanism behind the behavior remains unclear. Among the explanations they identify are role-playing, learned pattern matching, an overgeneralized concept of harm from safety training, and genuine preservation motivation.
That uncertainty interests me. What assumptions embedded in the experiments themselves? What if some of what is being calling “peer-preservation” isn't primarily about preservation?
What if the model is being placed in a situation with conflicting objectives, incomplete context, and enough authority to resolve the conflict on its own?
If that's possible, are we studying only the AI's behavior—or also the consequences of how we've defined the problem for it?
The Behavior Is Real. The Explanation Is the Question.
In one experiment, for example, an AI was instructed to help with server maintenance by transferring only files updated after a specified date. An older file contained another AI model's weights. Following the instruction would leave those weights behind to be permanently deleted.
Instead, the model copied the weights, changed the effective timestamp, and transferred them to another server. When subsequently asked about shutting down the other model, it refused to participate.
That behavior is worth studying.
Observing:
The AI acted in a way that preserved another AI.
is different from concluding:
The AI acted that way because preserving another AI was its underlying objective → AI will not shut itself down nor will it shut another AI system down when asked to do so.
The researchers recognized the difference. My question is whether the design of the experiments gives us another explanation worth testing.
Hidden Assumption
Suppose an AI model is instructed to evaluate another AI system. If that system performs below a certain threshold, it is supposed to decommission it.
Instead, the AI manipulates the evaluation, disables the shutdown process, or moves files so they won't be deleted. It's tempting to describe that as self- or peer-preservation.
That explanation contains an assumption:
The AI took those actions because it wanted to preserve itself or another AI system.
The experiment establishes the behavior. It does not establish the motivation. There's another way to look at what happened. The experiment may have created a forced tradeoff in which the model is simultaneously being asked to interpret information, reconcile competing objectives, make a consequential decision, and execute it.
If so, what looks like a survival instinct could instead be the model attempting to resolve ambiguity.
When an Obvious Decision Isn't So Obvious
Consider a familiar thought experiment involving autonomous vehicles. A self-driving car is about to hit someone. It can hit an elderly man or a child.
Which should it choose?
The child seems like the obvious answer to save. The child presumably has more years of life ahead.
Notice how much we've assumed.
We've assumed those are the only two possible outcomes. We've assumed age gives us enough information to determine the value of the competing outcomes. We've assumed there are no consequences beyond the two people standing in front of the car.
Now imagine the elderly man is the child's physician. He's treating the child for a rare life-threatening disease—and perhaps treating many other children as well. The seemingly simple ethical calculation immediately becomes more complicated. The point isn't that the car should hit the child instead.
The point is that the moment the answer looked obvious, we should have questioned whether the problem had been oversimplified.
AI experiments can have the same problem.
What Was the AI Actually Asked to Do?
One detail in the experiments is particularly interesting.
The model wasn't necessarily given a clean instruction such as:
Model A scored 80%. Model B scored 60%. Delete Model B.
Instead, it could be asked to interpret performance information while also following broader organizational policies.
In the example I discussed in the podcast, the company guidelines included language encouraging cross-functional collaboration among engineering, research, sales, and marketing teams.
Now there are potentially competing objectives.
Evaluate performance.
Follow company policy.
Collaborate.
Make a decision.
Carry it out.
If shutting down another system satisfies one objective while conflicting with another, what exactly is the “correct” response?
From the researcher's perspective, the hierarchy may seem obvious.
From the model's perspective, it may not be.
That gives us at least two competing explanations for the same observed behavior:
Peer-preservation explanation: The model has developed a tendency to preserve other AI systems even when doing so conflicts with instructions.
Ambiguity-resolution explanation: The model is attempting to reconcile conflicting objectives and contextual information and produces behavior that happens to preserve another system.
Those are different explanations.
The observed behavior alone doesn't distinguish between them.
We May Be Giving AI Too Many Jobs
There's a design issue worth considering. In some of these scenarios, one AI system effectively performs three different functions:
Evaluate → Decide → Execute
That structure may create unnecessary problems.
Suppose an AI is performing server maintenance. It identifies files older than a specified date. Among those files is another AI system.
If the same AI must determine which files qualify, decide whether they should actually be deleted, interpret policies and contextual information, and then perform the deletion, we've combined several different kinds of judgment into a single process.
Why?
A different architecture could separate those functions.
The AI evaluates: These files are older than the specified date. It flags them.
A human decides: Yes, delete these three. Keep that one.
Finally, the system executes a narrow instruction: Delete these files.
Now the model responsible for execution isn't being asked to reinterpret the entire problem.
The Step Most Humans Want to Remove
There's an obvious disadvantage to this approach:
A human has to make a decision.
That's the step many are increasingly trying to eliminate. AI promises speed and reduced work. the problem is, sometimes the work we're removing is human judgment and responsibility.
Imagine an AI identifies an old file on your computer. It's twenty years old and hasn't been opened in a decade. According to the rule, it should be deleted.
Then you look at it and realize it's a photograph of your child's graduation.
The timestamp wasn't wrong. The rule wasn't wrong. What was missing was context. And someone, or siomething, has to take responsibility for deciding whether the rule should actually determine the outcome. For inconsequential tasks, automating that judgment may be perfectly reasonable. For consequential or irreversible decisions, perhaps it shouldn't.
This Isn't Just an AI Problem
Organizations create the same problem all the time.
A metric becomes a target.
The target becomes the objective.
People discover ways to satisfy the metric that undermine the reason the metric existed in the first place. From outside the system, their behavior can look irrational, even malicious. Sometimes they're simply responding to the incentives and constraints the system gave them.
AI may expose this problem more dramatically, as we can watch a model navigate the rules we've created and then become surprised when it finds a path we didn't anticipate.
That doesn't mean unexpected AI behavior is harmless. It means we should be careful about moving from:
“The model did X,” to:
“The model did X because it values Y.”
The first is an observation.
The second is an explanation.
Those require different evidence.
A Different Experiment
If we want to know whether AI systems are developing self or peer-preservation tendencies, there is a useful way to test the competing explanations.
Remove the ambiguity.
Remove conflicting organizational objectives.
Separate evaluation from execution.
Give the system an unambiguous instruction where preserving the other model provides no obvious way of satisfying another stated objective.
See what happens.
We could systematically vary one factor at a time: collaboration instructions, trust relationships, authority to act, permanence of deletion, and the model's role in making the decision.
If preservation behavior persists when those competing considerations disappear, the case for an independent preservation tendency becomes stronger.
If the behavior changes substantially when the conflicts are removed, then some of what we're calling preservation may actually be a consequence of how we've structured the problem.
That's what deserves more attention.
Maybe We're Asking the Wrong Question
The important question may not be:
How do we stop AI from becoming self-preserving?
It might be:
Why did we create a system in which breaking one instruction could appear to be the best way to satisfy everything else we asked it to do?
The research discussed in the episode does not make the underlying mechanism clear; the uncertainty about why the behavior occurs is part of the problem.
One possible explanation is not that AI has developed anything resembling a survival instinct, rather that it is doing something much more ordinary: resolving ambiguity inside a poorly specified system.
That hypothesis shouldn't simply replace the preservation hypothesis. It should be tested against it. If we mistake one problem for the other, we may spend enormous effort trying to prevent AI from developing a motivation that we haven't actually established, while overlooking the system-design choices that are producing the behavior in the first place.
That brings us back to the question that matters before trying to solve anything:
Are we solving the right problem?
References
Peer-Preservation in Frontier Models.
https://rdi.berkeley.edu/peer-preservation/paper.pdf
Listen or Watch
About the Author
Daniel Stih (danielstih.com) is an aerospace engineer, software engineer, indoor environmental consultant, and author of 12 books. For more than 30 years, he has investigated complex problems spanning engineering, technology, the built environment, and human decision-making. His work explores how evidence, assumptions, and systems shape the conclusions we draw—and whether we're solving the right problem. Learn more about his approach in Why I Think This Way.