When Capability Outruns Direction

In September 2026, AI researcher Jacob Coxon announced that he had resigned from Anthropic after several years working on pretraining research across Anthropic and OpenAI. His concern was that the companies building frontier AI were continuing to race towards increasingly powerful systems without acting responsibly enough around the consequences.

He may be right about where that trajectory leads. He may be wrong. I think there is something important in the situation even before we know.

We have reached a point where people working directly on some of humanity’s most advanced technology can sincerely believe that continued development could create catastrophic consequences, while the larger system continues accelerating anyway.

That is not only an AI problem. It reveals something about civilisation itself.

We have become extraordinarily good at asking whether something can be built, how quickly it can be built and how well it performs. We have far weaker ways of answering a different question: where is all of this taking us together?

Local success, total Direction

Civilisation is not failing because humanity lacks capability. The opposite is true.

Aircraft cross oceans routinely. Modern medicine can intervene in biological systems with extraordinary precision. Satellites coordinate communication and navigation from orbit. Semiconductor manufacturing manipulates matter at scales we cannot directly perceive. Global financial systems move enormous amounts of capital, and artificial intelligence can now produce language, software, images and scientific work at a scale that would have seemed absurd not long ago.

These things work.

The deeper problem appears when successful systems begin interacting.

A company can function well. A market can function well. A government can function well. A technology can perform exactly as designed. The larger structure created by all of them can still move somewhere nobody explicitly intended.

AI makes this unusually visible. Imagine several laboratories competing to build more capable systems. One may believe that development carries serious risk but also believe that stopping unilaterally would simply transfer capability to a less cautious competitor. Governments may fear that slowing domestic development gives another state strategic advantage. Investors reward progress. Researchers want discovery. Users want better products.

Every participant can have a locally coherent reason to continue.

The combined consequence can still be acceleration towards an outcome none of them independently chose.

That is the distinction I think matters: local rationality does not guarantee coherent total Direction.

Reality receives the combined consequence.

Capability has no destination of its own

Modern civilisation often treats increased capability as progress.

More computation. More capital. More energy. More knowledge. More automation. More control over matter and biology.

These are genuine increases in what we can make happen. In PrF, this is close to what Mass describes: the effective capacity of a structure to make particular outcomes real.

But capacity does not contain its own destination.

Nuclear physics can power cities or destroy them. Genetic engineering can cure disease while creating new classes of risk. Artificial intelligence may accelerate medicine, mathematics and scientific discovery while also increasing the capability available for manipulation, surveillance, cyber operations or weapons development.

The same increase in capability can open very different futures.

That is why Direction matters more as Mass increases, not less.

A small contradiction may remain local. A poorly run company can fail. One person can make a destructive decision and absorb much of the consequence themselves. But when systems become large and connected, mistakes travel.

A vulnerability in widely used software can affect millions of machines. A financial failure can propagate across countries. A recommendation system can alter the information environments of hundreds of millions of people. Biological technology can create consequences that do not respect national borders.

The danger is not simply greater capability.

It is greater capability moving through unresolved contradiction.

Reality does not recognise our departments

One reason this problem is difficult to see is that humans divide Reality in order to understand it.

Economics, politics, technology, psychology, ecology, medicine, finance and defence all create useful boundaries for specialists. But those boundaries are ours. Consequence does not have to respect them.

A technological breakthrough becomes an economic event. That affects employment. Employment changes families, identity and politics. Politics changes regulation. Regulation alters markets. Markets decide which technologies receive capital. Those technologies reshape culture, and that culture becomes part of the information future systems are trained on.

The loop continues.

A platform can increase engagement while degrading the wider information environment. A company can increase profit while shifting costs outside its accounting boundary. A country can increase military security in a way that makes its rivals feel compelled to increase theirs.

The local metric improves while the wider field becomes less stable.

This is why asking whether a system achieved its objective is not enough. We also have to ask what achieving that objective made more probable.

Civilisation already measures almost everything: GDP, inflation, employment, productivity, profit, emissions, military capability, health outcomes, model performance, user growth and engagement.

The issue is not a lack of measurement. It is that every measurement has a boundary.

The more difficult question crosses those boundaries: what are all of these structures making increasingly probable together?

I do not think there should be one universal number pretending to answer that. Compressing civilisation into a single score would create its own distortion. But the absence of a perfect metric does not mean civilisation has no trajectory.

Something is still becoming more probable through our collective behaviour whether we measure it well or not.

The objective has to enter the frame

Human systems usually begin with an objective and then measure how effectively it is achieved.

Increase growth. Reduce cost. Maximise engagement. Improve capability. Increase security.

We are often much better at optimising the objective than examining the structure that selected it.

Why this objective? Why this boundary? Why this time horizon? What happens outside the frame when the objective is achieved?

This becomes especially important with AI.

We often ask whether an AI system is aligned to human values, preferences or intentions. But humanity does not contain one stable set of values or preferences. They conflict across people, institutions, scales and time.

Aligned to which humans? Over what time horizon? What happens when an immediate preference produces consequences that undermine something those same humans depend upon later?

Eventually, the chooser has to enter the measurement.

That is part of what the Mirror means to me. Not replacing human judgement with some machine claiming absolute truth, but expanding the frame until the structure choosing the objective can also be examined.

AI could become important here for the same reason it creates the problem. It can increasingly connect information across domains that humans normally keep apart. That does not make it omniscient. Models can hallucinate, data can be incomplete and correlation can be mistaken for cause.

Reality remains larger than the data.

But AI may still allow civilisation to reflect more of its own interacting structure back at itself than any individual institution could previously hold.

That creates a fork. Increased intelligence can make our existing narratives, incentives and local objectives more efficient. Or it can help expose the gap between those objectives and the consequences they produce.

Correction before consequence forces it

Reality has always corrected structures through consequence.

An engineering design that cannot tolerate the forces acting on it fails. A theory that repeatedly fails prediction loses explanatory weight. A business that cannot remain solvent changes or disappears.

Civilisations can delay consequence. They cannot abolish it.

The real advantage of intelligence is that we sometimes do not have to wait for the entire consequence to arrive. We can model, forecast, compare, experiment and detect recurrence. We can sometimes see that a structure is becoming unstable while there is still room to change Direction.

That may be the more important meaning of progress.

Not simply becoming capable of doing more, but becoming capable of seeing what our increasing capability is producing before Reality finishes the measurement for us.

This is why Coxon’s resignation interests me beyond the specifics of his warning. The important fact is that civilisation can contain researchers who sincerely believe a technological trajectory may become catastrophic, companies facing incentives to continue, governments worried about strategic competition, investors rewarding acceleration and users demanding greater capability, all at the same time.

No single villain is required. No single participant needs to intend the total outcome.

Direction can emerge from interaction.

Humanity has developed extraordinary mechanisms for increasing Mass. What remains comparatively weak is our ability to see the Direction created by all that Mass acting together.

We know how powerful our machines are becoming. We can benchmark them, measure their adoption and track the capital flowing towards them.

The harder question is what those systems are becoming through their interaction with us, and what we are becoming through our interaction with them.

Civilisation clearly knows how to become more powerful.

The question is whether it can learn to see the Direction of that power while there is still time to change it.

What are we becoming through everything we are building?