Progress in a capability can be steady while equivalence to it arrives all at once. The capability climbs a smooth curve, but whether it has caught up to some fixed reference is a single threshold, and the outcomes on either side of that threshold barely resemble each other.
- Performance against a fixed reference isn’t linear in the underlying skill; it’s closer to an S-curve. While you’re far behind, the reference wins almost every time, so years of steady improvement register as no change at all.
- It’s only in the narrow band around equivalence that each new increment starts to swing the result. Whatever is being overtaken doesn’t feel the long approach, only this last stretch of it.
- The trap is calibrating to the flat part. The slow approach feels like a wide margin, so you assume you have the time it took to get close, when what’s actually left is just the short crossing.
Horses
- Engines improved ~20% a decade for 200 years. For the first 120 the horses didn’t notice. Then 90% of them vanished in a single generation.
- Chess engines gained ~50 Elo/year for 40 years. The grandmaster who won 9 in 10 was, a decade later, losing 9 in 10.
- A horse was only ever wanted for one thing: traction, the literal unit of horsepower. So once something could do traction more cheaply, there was no second reason to keep twenty-five million of them around. There was no Jevons rescue either, since cheaper traction made us want more engines rather than more horses.
I very much hope we’ll get the two decades that horses did. But looking at how fast Claude is automating my job, I think we’re getting a lot less.
See also: Gall’s law, AGI, progress