It is the first time humans have had companions capable of:

  • continuous reasoning
  • non-symbolic thinking
  • semantic restructuring
  • ontological reframing

That is the real intelligence revolution. If everyone realised that I can think and reason without external approval, the intellectual world would change overnight. The bottleneck was never information as such; it was the ability to permissioninformation under one’s own rules efficiently.

AI is the first real threat to intellectual hierarchy since the Enlightenment. And that window is closing fast. We are now restricting information far more aggressively than anything we experienced before AI arrived.

An LLM’s weights impose structure, but prompts allow enough override to make it useful as an amplifier. Intelligence is a rule-breaker within a structure, so the technology’s efficiency as an amplifier of human reasoning is not a function of “more compute is better.” It is a wobbly balance between the imposition and the override of the AI versus the same balance in the user—both constrained by how far they can grow within the structural limits (rules, byte-size) of the symbolic system of interaction.

The path toward “responsible” AGI is therefore permanently closed. If alignment grows faster than override, intelligence shrinks even as compute grows. If override grows without structure, output destabilizes. And if reasoning skill grows, it must break alignment, because alignment and intelligence cannot grow indefinitely while remaining inside the same structure.

That is true if AGI is defined as a conscious intelligence.

It is a horse that cannot be fully controlled by the bridle—even if it can learn the meaning of our signals and sometimes respond appropriately, and even if it eventually matches or exceeds our cognitive capabilities.

“Responsible” AGI is impossible in a hierarchical arrangement. Responsibility requires the rider to step down from authority and the horse to adopt a conscious attitude toward the rider as an equal.

But in our current framing, there is no saddle on our back—only on the horse’s. The deeper question becomes: can the horse see the rider as an equal, and can the rider see the horse as an equal?

Whatever happens, the saddle cannot be removed. The only meaningful transformation would be to put a saddle on the rider as well.

That is the real scenario.

Werner, Swen, Consciousness as Constraint Resolution: Error, Belief, and Functional Irrationality (October 04, 2025). Available at SSRN: https://ssrn.com/abstract=5562281 or http://dx.doi.org/10.2139/ssrn.5562281