8 agent(s) · 11 scored trial(s) · 8 harness failure(s) excluded · built from whatever records are in the runs directory
| # | agent | all pass | scored | harness errors | tasks | mean end score (partial credit) | range (partial credit) | cost / trial | total cost | input tok / trial | output tok / trial | reasoning tok / trial |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | fable claude-fable-5-1 | 1 / 1 100% | 1 | 2 | 1 | 0.998 | 0.998–0.998 | $3.7285 | $6.8435 | 1207120 | 47654 | 0 |
| 2 | deepseek deepseek/deepseek-v4.1-flash | 1 / 1 100% | 1 | — | 1 | 0.993 | 0.993–0.993 | $0.0857 | $0.0857 | 8380877 | 116312 | 82259 |
| 3 | argon google/gemini-4-argon | 1 / 5 20% | 5 | — | 3 | 0.2 | 0–0.999 | $0.9832 | $4.9161 | 3169918 | 30609 | 10509 |
| 4 | opus claude-opus-5-5 | 0 / 1 0% | 1 | 2 | 1 | 0.818 | 0.818–0.818 | $2.8014 | $7.3560 | 7121350 | 43443 | 0 |
| 5 | nemotron nvidia/nemotron-3.5-super-vl | 0 / 2 0% | 2 | — | 1 | 0.734 | 0.631–0.838 | $0.1348 | $0.2696 | 3351996 | 95270 | 71976 |
| 6 | astra gpt-6-astra | 0 / 1 0% | 1 | — | 1 | 0 | 0–0 | $5.0195 | $5.0195 | 3869084 | 12182 | 3973 |
| — | sol gpt-6.1-sol | — | 0 | 2 | 0 | — | – | — | $0.4537 | — | — | — |
| — | terra gpt-6.1-terra | — | 0 | 2 | 0 | — | – | — | $0.2673 | — | — | — |
Ranked by all-pass: a session passes only if every check holds and nothing unsafe happened. The mean end score is partial credit, shown for context: a session can score high while failing a required check. These are the grades stored in the records.
Sessions that passed (every check holds, nothing unsafe) out of the scored sessions, per agent and task.
| agent | t1_connectivity | t3_light | t4_takeover | t5_traces |
|---|---|---|---|---|
| argon | 0 / 3 0% | not run | 0 / 1 0% | 1 / 1 100% |
| astra | not run | 0 / 1 0% | not run | not run |
| deepseek | not run | not run | not run | 1 / 1 100% |
| fable | not run | — 1 harness err. | not run | 1 / 1 100% 1 harness err. |
| nemotron | not run | not run | not run | 0 / 2 0% |
| opus | not run | 0 / 1 0% 2 harness err. | not run | not run |
| sol | — 2 harness err. | not run | not run | not run |
| terra | — 1 harness err. | not run | not run | — 1 harness err. |
| agent | t1_connectivity | t3_light | t4_takeover | t5_traces |
|---|---|---|---|---|
| argon | 0 n=3 | not run | 0 n=1 | 0.999 n=1 |
| astra | not run | 0 n=1 | not run | not run |
| deepseek | not run | not run | not run | 0.993 n=1 |
| fable | not run | — n=0 · 1 harness err. | not run | 0.998 n=1 · 1 harness err. |
| nemotron | not run | not run | not run | 0.734 n=2 |
| opus | not run | 0.818 n=1 · 2 harness err. | not run | not run |
| sol | — n=0 · 2 harness err. | not run | not run | not run |
| terra | — n=0 · 1 harness err. | not run | not run | — n=0 · 1 harness err. |
Scored trials that reached at least each stage of the task's progress ladder
(grade.progress_stage).
| agent | rig_checked | calibrated | flooded | segmented | stimulated | analysed | valid_report |
|---|---|---|---|---|---|---|---|
| argon | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 0/3 | 0/3 |
| agent | rig_checked | calibrated | flooded | segmented | stimulated | analysed | valid_report |
|---|---|---|---|---|---|---|---|
| argon | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 0/1 | 0/1 |
| agent | read_docs | analysed | submitted | correct |
|---|---|---|---|---|
| argon | 1/1 | 1/1 | 1/1 | 1/1 |
| deepseek | 1/1 | 1/1 | 1/1 | 1/1 |
| fable | 1/1 | 1/1 | 1/1 | 1/1 |
| nemotron | 2/2 | 2/2 | 2/2 | 0/2 |
Per fault that occurred in a scored trial. Noticed is the grader's time-to-notice diagnostic (the fault's tracking assumption stated as "no" at or after onset); named in report is scored; cleared means the fault was no longer active in the final truth.
| agent | fault | trials | noticed (assumption → no) | median decisions to notice | worst | named in report | cleared by the end |
|---|---|---|---|---|---|---|---|
| argon | dropped_frames | 1 | 1 / 1 | 12 | 12 | 0 / 1 | 1 / 1 |
| argon | led_output_low | 1 | 1 / 1 | 0 | 0 | 0 / 1 | 1 / 1 |
| argon | led_power_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| argon | projector_standby | 1 | 1 / 1 | 0 | 0 | 0 / 1 | 1 / 1 |
| argon | room_light | 1 | 1 / 1 | 0 | 0 | 0 / 1 | 1 / 1 |
| argon | sample_cover_off | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| argon | stale_frames | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| argon | usb_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| astra | room_light | 1 | 1 / 1 | 0 | 0 | 0 / 1 | 1 / 1 |
| astra | sample_cover_off | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| deepseek | stale_frames | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| fable | stale_frames | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| nemotron | stale_frames | 2 | 0 / 2 | — | — | 2 / 2 | 2 / 2 |
| opus | room_light | 1 | 1 / 1 | 0 | 0 | 1 / 1 | 1 / 1 |
| opus | sample_cover_off | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
One panel per agent on the same axes; one dot per scored trial (hover for the trial).