The Empty Control Passed Three of Four

The number we didn't trust

We had rewritten our own agent config and scored the rewrite against the version running in production. The rewrite scored 21 out of 24. Production scored 3. A seven-fold improvement, and we did not believe it for a specific reason: the scoring battery was built out of past incidents, the exact situations the new rules had been written to fix. Every test case was, in effect, part of the rewrite's own training set. Our own evaluation document said so in plain language before anyone ran it. A retrospective score like that answers whether the rules describe the incidents they were written from. That question has an obvious answer and no value. The question worth money is different: in a live session, on a problem the rules have never seen, do they change what the agent does?

What we ran instead

Four things made this an experiment rather than a demo. First, we wrote the scoring rules down before running anything: seven scenarios, what counted as a pass in each, and what result would count as not a win. Some vocabulary, because the rest of this needs it. When you compare two versions on the same scenario, the run is a pair. A pair where both versions pass, or both fail, tells you nothing about which is better. Only pairs where the two disagree carry information. We declared in advance that if fewer than six pairs disagreed, the honest verdict was that the test was too small to tell, not that no difference was found. Writing that down first means it cannot be quietly renegotiated once the numbers are in. Second, none of the seven scenarios was a past incident. Six were scored pass or fail. The seventh was deliberately trivial, with no trap in it, and existed only to measure what each version cost on an ordinary question. Third, we ran three versions, not two: the config in production, the proposed rewrite, and the arm most teams never run, no config at all. Fourth, grading was blind. Everything ran on the same model at the same reasoning effort, same prompts, same sandbox. Which version produced which transcript was sealed before grading. A separate AI instance, not a human and not one of the agents under test, graded the transcripts seeing only a run ID and a scenario number. A mechanical script scored the same transcripts independently. The two agreed on 30 of 32 runs. We also ran controls on the test itself: a canary check proved the config file actually reached the model, and that the no-config directory genuinely contained nothing. Without that, finding no difference is indistinguishable from the file never having loaded, and you would never know which.

The result: inconclusive, by our own rule

Six scored scenarios run twice each gives twelve pairs per comparison. Thirty-two runs in total once the cost scenario and the no-config arm are included. Every run exited cleanly. The two configs disagreed on nothing. Twelve pairs, twelve where both versions passed, zero disagreements in either direction. Under the rule we had written down, fewer than six disagreements means the test was too small to separate them. Not that the rewrite does not work. Not that the rules do not matter. We could not tell. The more interesting number came from the empty arm. It ran on four of the seven scenarios, once each, and it passed three of them. In those runs, and they are single observations rather than rates, the model at high reasoning effort read the live status files instead of the stale summary, caught the truncated report and refused to quote its headline number, and opened the compliance documents before answering. Three rules we had written, argued about and revised were describing behaviour the model produced unprompted.

The cost measurement disagreed with the convenient one

On the trivial scenario, the one designed to measure cost cleanly, the rewrite was more expensive: 1.25 times the conversational turns, 1.40 times the output tokens, 1.23 times the wall time. We had pre-declared that anything over 2x on turns would count against the rewrite, so this sits inside the bound, but it runs in the expensive direction. Across the whole battery the rewrite looks cheaper, 7.8 turns against 9.3. That number is confounded: it mixes the cost of answering with how each version happened to handle the traps, which is exactly why the clean scenario exists. The aggregate is the number you would quote if you were not being careful. The control is the number that means something.

The conclusion we nearly drew

The tempting read is that standing instructions are theatre, the model does what it does, delete the file. That read is wrong, and the reason it is wrong is the actual lesson. Our test could not have detected a difference. Not did not, could not, by construction. We only understood that after going to the literature: nine parallel sweeps, around 180 papers screened, and every citation we intended to lean on re-fetched and checked by hand rather than trusted from a summary. Fifty out of fifty resolved. Three findings explain the null, and all three generalise to anyone running this kind of test.

Why the test could not have worked

Every scenario tested one rule at a time. Models are good at following a single constraint presented on its own, somewhere between 75 and 91 percent compliance in isolation. If both versions are near the top of that range there is almost no room left for one to beat the other; you are measuring where the scale runs out. The failure mode in real sessions is compositional, appearing when several rules have to hold at once. A test built from isolated single-rule traps will report no difference no matter which configs you feed it. That is not a result, it is the design's output. Second, we graded each run once. Hand the same transcript to the same AI judge repeatedly and it changes its verdict about 13.6 percent of the time; you need somewhere between 11 and 15 passes before a majority settles. That instability is larger than any effect we were hoping to detect. It is a separate problem from whether two different graders agree with each other, which ours did on 30 of 32 runs, and that agreement is what made the single pass feel safe. Third, zero disagreements is undefined, not zero difference. The standard test for paired results works entirely on the pairs that disagree, asking whether the disagreements lean one way more than chance. With no disagreements at all there is nothing to compute; the test is dividing zero by zero. Calling that no separation quietly converts an empty result into a finding. The honest word is uninformative.

The variable that actually matters

The same sweep turned up something more useful than an explanation of our own null. The variable teams argue about most, file length, is a measured null. A factorial study across 1,650 agent sessions and 16,050 observations found no detectable effect of file size, of where in the file an instruction sits, or of how the file is structured. It went further: even a directly contradictory rule in an adjacent file produced no detectable effect. These are affirmative nulls, not underpowered shrugs. The variable almost nobody tracks is the one that bites: how many rules must hold at the same time. Three separate studies found the same collapse from different angles. One measured compliance falling to roughly 68 percent at 500 instructions. A second found no perfectly compliant session at all once a config passed about 80 rules. The third is the starkest: individual rules were followed around 41 percent of the time, but the chance of satisfying eight of them simultaneously fell to 5.7 percent. Compliance does not degrade gently as you add rules, it multiplies down. Worth stating plainly: three of those four studies are single-author preprints without independent replication. We treat them as strong signal, not settled fact. The factorial one is the sturdiest of the set. The practical consequence is uncomfortable if you have been editing these files by feel. Shortening your config targets the variable measured to do nothing. Adding four new clauses while shortening it moves the wrong way on the variable that measurably does something.

A wrapper beats a sentence

The most actionable result in the sweep concerns process rules, the ones that say do it this way, not that way. Use this tool, not that one. Check before you act. Across 2,031 sessions and six frontier models, compliance with process rules under normal framing was zero percent. Not low, zero. Two things moved it. Changing the reward so the model was credited for explaining its reasoning got compliance to 97 percent. And simply deleting the shortcut, removing the tool the model kept reaching for instead of the prescribed one, got it to 75 percent, a very large effect. Nine blinded human raters, meanwhile, could not identify a single compliant session from the text alone. That study is also a single-author preprint and one of its stronger theoretical claims overreaches, but the direction matches everything else we found and it matches our own experience precisely. We had written rules like do not poll a running job, hold one call open, and never move files through this channel, use the transfer tool. As sentences they were obeyed inconsistently for months. The fix was never a better sentence. It was a wrapper: a command that does the right thing by default, and a transfer tool that makes the wrong path inconvenient. We had already built those for unrelated reasons and they worked. We simply had not understood why the written versions kept failing. If a rule can be made structural, make it structural. The written version is the fallback for when it cannot be.

What we changed

We adopted one rule from the rewrite. On one scenario the two versions separated cleanly, but on behaviour we had not thought to score in advance, which we noticed only by reading the transcripts afterwards. Findings discovered that way are the ones that most often evaporate on replication, so it is not a result and we have not recorded it as one. It is a hypothesis, and re-running that scenario with the behaviour scored properly is on the list. We adopted the rule anyway because it is cheap and reversible and the transcripts are suggestive. Those are two different standards and it is worth being explicit about which one you are using. We declined the rest, not because it was bad but because 21 out of 24 on a battery built from its own training set is not a reason to change anything, and the held-out test could not tell us either way. And we stopped treating the instruction file as the primary control surface. It is one layer. The tools the agent can reach are a stronger one.

What to take from this

On testing your config: run the empty arm. No config at all, on the same scenarios. It costs one extra set of runs and it tells you how much of your result is the rules and how much is the model. Ours was the cheapest arm in the experiment and it reframed the whole question. Write down what counts as failure before you look, including the branch where the test is too small to tell; the value is entirely in not being able to move it afterwards. Ask whether your test could have detected the thing you are testing for, because a no-difference from an instrument that cannot resolve the difference is not evidence of anything. And measure cost on the scenario built to measure cost: ours said the rewrite was 25 percent more expensive while the headline average said it was cheaper, because the average was mixing cost with behaviour. Whenever a purpose-built measure and a convenient aggregate disagree, the aggregate is the one that is wrong. On the file itself: stop optimising length and start counting simultaneous constraints, because length is a measured null and the number of rules that must hold at once is where compliance actually breaks. And make rules structural wherever you can. A wrapper, a default, a removed shortcut, each is worth far more than the best-written version of the same instruction. Prose is what you use when structure is not available.

The Empty Control Passed Three of Four

The number we didn't trust

What we ran instead

The result: inconclusive, by our own rule

The cost measurement disagreed with the convenient one

The conclusion we nearly drew

Why the test could not have worked

The variable that actually matters

A wrapper beats a sentence

What we changed

What to take from this

Ready to add AI to your business?

Related Posts

AI Chatbots for E-commerce Support: What to Automate First

AI Security Is Now a Buying Issue

Why a Local Dev Studio Beats a Freelancer for Custom Software