OSCILLA®

RESEARCH NOTE RN-014 · VERIFICATION GROUP · 04 NOV 2026

Counter-example search at 1.2 million cases: breaking a proof before it ships

Every proof Oscilla ships is first handed to a search whose only job is to break it. This note reports the largest run of that search to date — 1.2 million attempted counter-examples across 4,800 production proofs — what it found, and the three counter-examples that changed the roadmap.

AUTHOR  JONAS REBER

GROUP  VERIFICATION

READING TIME  12 MIN

STATUS  PUBLISHED IN THE NOTEBOOK

01A claim, not a trophy

A finished proof is usually treated as a trophy: the run converged, the steps check, the answer ships. We treat it as a claim — a statement about the world that might be false in a way the proof itself cannot see. A checker verifies that each step follows from the last. It does not ask whether the first step described the problem that was actually posed.

That gap is where production failures live. In our incident reviews, the reasoning was almost never wrong mid-proof; it was wrong at the boundary — a unit taken as another unit, a quantifier read one way by the model and another by the engineer, a tool result trusted past its scope. None of those are step errors. All of them are breakable — if something goes looking.

So something goes looking. Before a proof counts as shipped, a second process — the counter-example search — receives the claim, the premises and the proof, and spends a fixed budget trying to construct a case where the premises hold and the conclusion does not.

02How the search runs

The search works the boundary first. It mutates the premises at their edges — boundary values, empty and maximal sets, units and frames — then works inward, weakening one assumption at a time and re-running the checker on each mutation. A surviving mutation is a counter-example: premises true, conclusion false, proof formally intact and practically wrong.

DEPTH 6 — 92% FOUND DEPTH → CASES ATTEMPTED → 1.2M FOUND
FIG. 31 — SEARCH EFFORT BY DEPTH: NEARLY EVERYTHING BREAKABLE IS BROKEN BY DEPTH SIX · DEMO DATA

The budget is the policy: every production proof gets the same depth, the same mutation families and the same wall-clock, whether it was commissioned by a customer or by our own roadmap. The policy that ran for this note is short enough to print.

LISTING 1 — THE SEARCH POLICY, AS RUN

search:
  target:        claim + premises + proof
  budget:        depth 8 · 250 cases per proof
  mutate_first:  boundaries, units, empty sets
  then:          weaken one premise at a time
  stop_when:     a case survives the checker
  on_survival:   proof is returned, not shipped
  record:        every case, kept in the run ledger

03What 1.2 million cases found

Of 4,800 production proofs, the search broke 151 — 3.1 per cent. Almost all of them broke early: the median proof that fell, fell within its first 1,204 attempted cases, and 92 per cent of all counter-examples were found by depth six. Depth is cheap. Not looking is what costs.

The breaks were not evenly spread across the proof. Grouped by the step the counter-example indicted, they gather in the middle — the steps where the model stops restating the problem and starts trusting its own intermediate results. Step four is where proofs go to be believed too early.

S1 S2 S3–S4 S5 S6 S7+ THE BELIEVING STEP
FIG. 32 — COUNTER-EXAMPLES BY THE STEP INDICTED, PER 1,000 PROOFS · DEMO DATA
TABLE 1 — REVIEW ONLY VS. REVIEW + SEARCH, TWO RELEASES · DEMO DATA
MEASUREREVIEW ONLYWITH SEARCH
Proofs shipped with a surviving counter-example90
Proofs returned for rework per release21238
Median time added per run—41 S
Counter-examples kept in the ledger0151

“A proof that survives a deliberate attempt to break it is a different object from a proof that was never attacked. We ship the first kind.”

JONAS REBER — HEAD OF VERIFICATION

04The counter-examples that mattered

Most counter-examples were housekeeping: a premise stated loosely, tightened in an afternoon. Three were not. One found a load case the premises described in kilonewtons and the proof consumed in newtons — every step valid, the answer wrong by three orders of magnitude. One found a scheduling claim that held for every week except the one containing a public holiday. And one found a proof whose third step quietly assumed the thing the sixth step was supposed to establish.

THE HOLIDAY WEEK CLAIM AT THE ROOT — LEAVES CHECKED, ONE LEAF FALSE
FIG. 33 — THE THIRD COUNTER-EXAMPLE, AS A TREE: EVERY LEAF HOLDS BUT ONE · DEMO DATA

Each of the three is now a permanent test. The ledger keeps every counter-example the search has ever found, and every model release re-runs all of them before it re-runs anything else. A failure mode, once seen, is never allowed to become invisible again.

05What changed in Oscilla-1.2

Three changes shipped with 1.2, all downstream of this run. The search is no longer a gate at the end of a run but a companion to it: from step four onward — the believing step — mutations are attempted while the proof is still being written, when redirecting it is cheap. Units and frames are declared in the premises in a form the checker can enforce, so the kilonewton case now fails at the first step instead of surviving to the last. And run replay reproduces a search exactly, case for case, so a counter-example found once can be re-examined by anyone, forever.

The deeper change is procedural. A proof is no longer described in the console as verified; it is described as attacked, depth 8, survived. The words matter. They keep the claim a claim — and keep someone, or something, always looking for the case that breaks it.

WRITTEN BY

Jonas Reber

Head of Verification at Oscilla Labs. Jonas built the counter-example search after a career in formal methods for rail signalling, where the counter-examples are not hypothetical. He still reads every one the search finds.

RELATED — FROM THE NOTEBOOK

RESEARCH · OCT 2026

Where proofs break: failure rate by reasoning step

READ THE NOTE →