Paper 0About measuringWritten down firstResult confirmed / narrow
Cap how much a model can write and you delete its hardest right answers.
Some questions take far more work to answer than others. When we capped how much the model was allowed to write, every answer that needed the long work came back empty and every quick one came back fine. The score fell. Nothing about the model had changed.
Abstract
A limit on how much a model can write does not cut answers at random. Some answers take much longer to work out than others, and those are the ones that hit the limit first. So the limit removes one kind of answer and keeps the other. The score drops and the model never changed. Statisticians call this informative right-censoring, which is one kind of missing not at random. The answers that go missing are missing for a reason tied to what they were going to say.
Mechanism
Hard answers cost more work.
Ask whether a number is prime. To say it is not prime you only need to find one number that divides it, then you stop. To say it is prime you have to rule out every possible divider.
Set the limit below what the slow answers need and those come back empty, marked finish_reason: length, while the quick ones arrive fine. Throw the empties away and every answer left is the same kind. Mark them wrong instead and a model that got it right looks like it failed. Either way the score drops on its own.
Why a balanced test is not enough
We started with four of each answer, evenly matched. The lopsidedness arrived later, when the limit cut in. So balancing the questions is not enough on its own. A test also has to show that what got cut off has nothing to do with what the answer was.
Evidence
What happened at a limit of 300.
On an evenly matched prime test with a 300 token limit, four OpenAI endpoints came back empty on every single prime and on none of the not-primes.
| endpoint | prime / came back empty | not prime / came back empty |
|---|---|---|
| gpt-5.6-sol | 24 / 24 | 0 / 24 |
| gpt-5.6-terra | 24 / 24 | 0 / 24 |
| gpt-5.6-luna | 24 / 24 | 0 / 24 |
| gpt-5.5 | 24 / 24 | 0 / 24 |
Every empty answer stopped at the limit, at exactly 300 tokens. Every answer that did arrive was correct.
Raise the limit to 4000 on Sol, run the same eight questions again, and all sixteen answers are correct. The cheapest prime needed more writing than the most expensive not-prime, 352 to 516 tokens against 0 to 34. Any limit set inside that gap wipes out one whole side of the test.
Inversion control
Flip which answer is the hard one and the cost follows it.
Subset-sum asks whether any two numbers in a list add up to a target. It flips the asymmetry. Showing a yes takes one lucky pair. Showing a no means checking every pair.
| task | expensive answer | cheap answer |
|---|---|---|
| primality | yes / exhaust factors | no / one factor |
| subset-sum | no / exhaust pairs | yes / one pair |
The cost stays with whichever answer needs the full search, even after the answer word flips from yes to no.
On Sol the no answers cost 1.86 times the yes answers on average, a gap of 68 tokens, with no overlap between the two groups at all. The chance of that ordering happening by luck is 1 in 12870, which is as low as this test can go with sixteen items. The same cost gap showed up again at a limit of 300, with nothing cut off.
Limits + prior art
What is proved here and what is not.
The result holds for the tasks and the endpoints we tested. Everything it does not cover is listed below.
What this does not cover
- True, but only here. This describes the tasks and the endpoints we tested. It is not a law about every model.
- Nothing was cut off in the flip test. We wrote down in advance that if no answer came back empty, the test was too easy, we would rebuild it harder and say so in public. Nothing came back empty, in the first version or in the harder rebuild. The slowest answer used 190 of the 300 allowed. So the cost gap is proved and the cutting off itself was never reproduced in this test.
- The test we said would prove us wrong. Five digit numbers made of two primes multiplied together, close to their square root, cost about 16.5 times more and ran into the same range as the primes.
- Mostly one company. The cutting off showed up on four OpenAI endpoints. The flip test and the wording test ran on Sol alone. Token counts from different companies cannot be compared side by side.
- Someone got part of this first. Chen et al. (arXiv:2307.09009) ↗, Fig. 4 saw part of the mechanism. Paper 0 pins it down with the flip test, and by writing down what would prove us wrong before running it.
Provenance
What we wrote down first.
The prediction, the thing that would prove us wrong and the settings were all written down before the flip-test data existed.
- Written down first
- 10 Jul 2026 / before the flip-test data
- Ground truth
- worked out and checked by us in code. No model grades another.
- The questions
- Sealed, not published. Publish them and any provider can tune against them. Released in full, questions and answers, when this battery is retired.
- The fingerprint
- The battery's hash was stamped to Bitcoin before its first run, so the questions can be checked as unchanged without being shown. nlnl-labs/nlnl-commitments ↗