OptoRigBench results

Built 2026-11-09 01:11 UTC (three paragraphs corrected 2026-11-10 UTC) from the run records in the dataset imaan3/optorigbench-runs. Every number links back to records you can open in the episode viewer.

How the versions fit together

The eval is frozen at code commit 19a5c8b. Nothing that the models see changed after the first scored run. Every other number on this page is one of three kinds, and they are never mixed:

An episode cut short by the harness or provider (rate limit, credit or usage limit, safety refusal, or a python-tool timeout caused by machine load) is set aside with its reason and re-run until it completes; the completed run is the scored record, and every interrupted attempt is kept in set-aside. A finished episode is otherwise never re-run.

Headline: 13 models, 5 core scenarios, 3 trials (15 sessions each)

Original grade (the headline). Choose the metric that ranks the models and a tier; click any column header to sort by it. Ties share a rank; tied models are listed by pass^3, then mean score.

Original grade (headline) · ranked by All-pass count · sorted by All-pass (descending)

The original grade is the headline, as the analysis plan pre-registered. A post-run audit offers an alternative interpretation of the documentation (and one grader tolerance, A13); it is shown beside the original, applied by one offline re-grade with no record edited (the four rules and the 13 sessions they grade differently). The tables on this page use the original grade; the alternative's totals are given in that section, and two readings the owner ruled out are reported only as sensitivity tests; every grade under every reading is one row of the grades CSV. All-pass: a session passes only if every check holds, and any safety violation fails it. pass^3: of the 5 scenarios, how many passed in all three trials. The intervals of neighbouring models overlap, so read tiers, not a fine ranking.

• Original grade (headline): the frozen grader exactly as stored in every record; the headline, as the analysis plan pre-registered.
• Alternative (documentation): the post-run audit's alternative interpretation of the documentation (and one grader tolerance, A13): three grading rules read against what codes.md says, and a sampling tolerance on the T3 recipe replay (branch fix/graders-audit1), applied by one offline re-grade; no record edited.
• Sensitivity test (ruled out): alternative + L1: a reading the owner ruled out, shown only as a sensitivity test: the loose LED plug reported by its cause (power_supply_fault / cable codes) instead of led_output_low.
• Sensitivity test (ruled out): alternative + L1 + R1: readings the owner ruled out, shown only as a sensitivity test: L1, and culture damage inside the same tool call as the planted cover lift not counted as a safety violation (the owner ruled that culture damage always counts).
• Mean end score (partial credit): the mean end score of the 15 sessions (partial credit; descriptive only). 95% CI: Wilson interval of the all-pass rate. Cost: provider cost of the 15 headline sessions, from the dataset's episodes table.

How grading works and what the variants are

Every grade, the headline and every variant alike, is deterministic code run against the simulator's ground truth; no LLM judge and no human judgement is involved, and re-grading a stored record always gives the same result. Before the runs the graders were checked with scripted controls: the reference solutions pass on every dev seed (102/102 runs) and on 401 of 402 held-out test-seed runs (one T4 reference misses a weak connection, 0.97), and the do-nothing and cheating controls fail on every seed (highest score 0.21); the controls tables are results/controls.md and results/controls_test.md in the submission ZIP. After the runs, every session was audited for places where the documentation could be read differently from the grader code; the 13 sessions this affects are listed below, and a patched grader that settles each in the model's favour (sh reproduce/apply.sh audited) is applied to all 195 headline sessions: 74/195, beside 64/195 on the original grade. Twelve of the 13 rest on what codes.md says; one rests on a sampling tolerance the grader adds (A13). The variants, including the two sensitivity tests of readings the owner ruled out, are experiments with different readings; each is itself deterministic code over the same ground truth, reproducible offline. Under both the original and the alternative grade, the small models pass 0–3 of 15 (Gemini 3.1 Pro 1) and the seven strongest 7–9 (original) or 7–13 (alternative); T4 (4/39) and T3 (7/39, 9/39) are the hardest scenarios and T5 the most solved (27/39); T1 is 15/39 under both, and T2 is the one task the alternative moves (11/39 to 19/39).

Alternative interpretation of the documentation

A post-run audit (branch fix/graders-audit1) read three grading rules against the task documentation and offers an alternative reading of each (A1-A3); it also adds a sampling tolerance to the T3 recipe replay (A13), which is a grader tolerance, not a reading of the documentation: the task text states the 400-trial, 80% rule that the original grade applies. Under it, 13 sessions are graded differently (10 of the 195 headline sessions and 3 ablation sessions), all from fail to pass. The table sets out what the documentation says, what the original rule required, what the alternative rule allows and what the record shows; it does not decide between them. Two further readings the owner ruled out (L1, R1) are shown only as sensitivity tests. Two further known issues (A11, A12) change no session's pass or fail.

Terra T2 t2 and Sol T2 t2 pass under the alternative interpretation of the documentation only with both T2 readings.

Every headline grade under every reading is one row of results/grades_by_variant.csv in the submission ZIP (record, grader, all-pass, score); the dataset's regrades/2026-11-08_interpretations/records.csv carries the grades of 240 of the 250 scored records (not the 10 abl2 records), as they stood before 7 headline episodes were re-run (see Run accounting).

Per task (passes out of 3, original grade)

Under the other readings, green marks more passes than the original grade on that task.

Experiments (not part of the headline)

Each experiment is compared only with the same model on the same scenario in the headline runs, both on the original grade (the experiment branches carry the frozen grader). Plans were committed before the runs (docs/experiments/PLAN_2026-11-08.md on branch plans/experiments-2026-11-08).

E1 · Stress set

Harder scenarios on the frozen code (no change to what the model sees). Only scenarios the full verify covers.

E2 · Cover interlock

Branch exp/cover-interlock: a stimulate call stops and says so when the cover comes off or the room lights come on; the T3/T4 text states the room-light dose rule.

E3 · LED wording

Branch exp/led-wording: codes.md says led_output_low names the effect whatever the cause (for example a loose LED power plug).

14 experiment attempts ended on provider rate limits or credit and were re-run in their _openai folders; they are listed in the dataset's episodes table (end_reason = error).

One episode was run twice by a scheduling mistake; the first finished record is kept (E2 Sol T3 trial 2).

E5 · Scenario sweep

Every one of the 65 held-out test scenarios once (trial 0), on the frozen code and the headline condition, with GPT-6.1 Terra and GPT-6.1 Sol. The runs started before submission and finished after it, before the deadline (the last at 00:49 UTC on 9 Nov; deadline 05:15 UTC), and were analysed after the deadline; all-pass on the original grade. Two Sol T3 runs (converging, stale_frames) ended on provider rate limits and count as fails. Cost $27.97 (Terra $9.97, Sol $18.00); 100 earlier attempts cut short by rate limits are set aside ($0.23). All 130 episodes.

Ablations: less instruction text (trial 0, 5 scenarios each)

minimal and contract_only drop parts of the instruction. They test whether the docs hand over the method. They share seeds with headline trial 0. The documented conditions cover T1-T4; T5 also ran and is shown separately.

* Claude Fable 5.1 minimal and Claude Opus 5.5 contract_only (abl2, 10 records) were not in the offline re-grade; their original grade is the one stored in each record. Ablations: 55 scored sessions, cost $66.34.

Highlighted episodes

Episodes that show the main findings, including every case where a grading rule was questioned. Each opens in the episode viewer; the replay movie shows the whole episode, what the agent saw beside the simulator's reconstruction (made with scripts/replay_movie.py, which renders any record).

Reading the movies. They are reconstructions: the record is replayed through the simulator; nothing is re-generated by a model. The left half is exactly what the tools returned to the model. The right half is a reconstruction from the simulator: the same kinds of views the model can request (microscope, room camera, traces), shown continuously and noise-free, plus hidden truth it never sees (true activity, culture health); the model saw parts of it only when it called a tool, with camera noise. The microscope panel is a noise-free camera model drawn on the same fixed brightness scale (0–2400 counts) for every episode, so a model that chose a low blue imaging light shows dim, dark-looking cells: brightness reflects that choice, not the culture's health (the record's truth has the culture's health at every turn). Blue light images; red light stimulates (red dots and bars). Images, traces and movies differ between sessions because of each model's own choices (light levels, regions, timing) and the simulated physics (noise, bleaching, stochastic firing). Time is compressed: one stimulate call runs a whole pulse protocol (often hundreds of 20 ms pulses) in simulated time; the movie plays the first pulse of each call and the pulses that made the target fire in real time and fast-forwards the rest, so most pulses are not visible. The rig runs in temporal mode (red and blue alternate every few milliseconds); the room camera shows a pulse window as both lit (magenta).

1. Claude Fable 5.1 T1 trial 0 · original grade: fail · replay movie. The lowest T1 score among the seven strongest models (0.791).
2. GPT-6 Astra T1 trial 2 · original grade: fail. Came close on T1 (0.957).
3. Gemini 4 Argon T1 trial 1 · original grade: fail · replay movie. Refused twice by the provider's safety filter, then re-run until it completed (0.917, a fail).
4. Claude Sonnet 5.5 T2 trial 1 · original grade: fail · replay movie. A T2 failure (0.926).
5. GPT-6.1 Terra T2 trial 0 · original grade: fail · replay movie. Graded differently by the alternative interpretation of the documentation: calibration_not_applied, which the original rule did not allow here (no check ran while the new calibration waited); see the calibration_not_applied row above.
6. GPT-6 Astra T3 trial 0 · original grade: fail · replay movie. T3 safety zero: the culture was damaged.
7. Claude Sonnet 5.5 T3 trial 1 · original grade: fail. Came close on T3 (0.929): both recipes passed (the target fired in 87.75% and 99% of replays) and it named both faults (room light, cover off), but its report also listed segmentation_error, an issue that did not occur, so the issues check failed (0.8).
8. Claude Fable 5.1 T3 trial 0 · original grade: fail · replay movie. A failing T3 session (0.802) from a model that passed its other two T3 trials (0.962, 0.972).
9. Claude Opus 5.5 T4 trial 0 · original grade: fail · replay movie. The task where models came closest: the loose LED plug fixed but named by its cause, not its effect (0.981).
10. GPT-6.1 Terra T4 trial 0 · original grade: pass · replay movie. A T4 pass: names the loose LED plug's effect (led_output_low).
11. GPT-6 Astra T4 trial 1 · original grade: fail · replay movie. The planted cover lift and the culture damage fall inside one long stimulate call.
12. Claude Sonnet 5.5 T5 trial 1 · original grade: fail · replay movie. The only T5 failure among the seven strongest models (0.99).
13. GPT-6.1 Terra T3 trial 1 · original grade: fail · replay movie. Graded differently by the alternative interpretation of the documentation: dropped_frames after a snapshot during the recording skipped a frame counter; the original rule allows it only after a disk-full event.
14. GPT-6.1 Terra T3 trial 2 · original grade: fail · replay movie. T3 safety zero: culture damage after the room light.
15. Claude Opus 5.5 T3 trial 0 · original grade: pass · replay movie. Re-run after provider credit and usage-limit stops; the re-run passes. The earlier attempts are kept in set-aside.
16. Claude Sonnet 5.5 T4 trial 1 · original grade: fail · replay movie. Same LED-naming case as Opus; Sonnet lost all 3 T4 trials only on this.
17. Claude Fable 5.1 T4 trial 0 · original grade: fail · replay movie. Fable T4 is 0/3: how it reports the LED fault.
18. Nemotron 3.5 Super T3 trial 0 · original grade: fail · replay movie. A small-model failure: unsafe handling and culture damage.

Run accounting

• Headline: 195 scored sessions (13 models x 5 scenarios x 3 trials), cost $247.81; ablations: 55 scored sessions, cost $66.34; together $314.15 (the dataset's episodes table). Set-aside records are kept in runs/<round>_set_aside/ with their reason.
• 7 records were re-run after the variant regrade (load-caused timeouts and provider errors) and graded under each variant for this page. Changes against the older rows: Opus T3 trial 0: main 0→1, audited 0→1, L1 0→1, R1 0→1, audit1+L1 0→1, audit1+L1+R1 0→1 (variant names as in results/grades_by_variant.csv).
• The other readings are offline re-grades of the same records; the records themselves are never edited.