There’s a particular kind of excitement around AI that fixates on the ceiling — how clever can these systems get, how many exams can they pass, how close to some threshold of general capability are we this quarter. It’s the part that makes headlines. It’s also, I think, the least interesting question we could be asking.

The question that actually decides whether any of this is good is quieter: can a person tell what the system did, and why, and stop it when it’s wrong?

Call it legibility. It’s the property that lets a human look at a machine’s behaviour and understand it well enough to trust it, audit it, or overrule it. And it is consistently undervalued, because it doesn’t photograph well. Nobody livestreams a keynote about confirmation dialogs. But legibility is the difference between a tool you can hand real responsibility to and a very impressive black box you have to either blindly trust or refuse to use.

Confidence without checkability is the dangerous part

A capable model that acts confidently and opaquely is genuinely dangerous in a way a weaker, transparent one is not. If a system tells you it has completed a task and it is wrong, the cost of that confidence is borne entirely by you — and you have no way to catch it unless the system around it was built to show its work. “Trust me” is not a feature. It’s the absence of one.

This is why the unglamorous engineering is the compelling part: the verification step, the read-back, the action staged for review instead of fired off, the explicit line between reading and changing. These aren’t bureaucratic friction slowing down the cool part. They are the cool part — they’re what converts raw capability into something a person can actually rely on. An agent that can do a thousand things but can’t show you which one it just did is less useful than an agent that does ten things and tells you the truth about each.

Constraints are what make capability spendable

There’s a temptation to frame guardrails as the opposite of capability — as if every safety check is a tax on what the system could otherwise achieve. That framing is backwards. The constraints are what make the capability spendable. You will give a careful system more latitude, not less, precisely because you can see what it’s doing. Freedom for an AI agent isn’t the absence of checks; it’s the trust earned by being checkable. The most autonomous systems will be the most legible ones, because those are the only ones anyone sane will let off the leash.

Legibility has a moral dimension worth dwelling on. When a system makes a decision that affects someone — what they’re shown, what they’re offered, what they’re flagged for — the person on the receiving end deserves a system whose reasoning can be surfaced and contested. An unaccountable model making consequential calls isn’t efficient; it’s just unaccountability with better PR. The right to an explanation isn’t a nice-to-have bolted on at the end. It should be load-bearing in how these things are designed.

None of this is an argument against ambition. These systems should be capable — enormously so. But the binding constraint on whether that capability becomes something good is not how smart the model is. It’s whether the humans in the loop can see, verify, and steer. The boring part was never boring. It was the whole point.