5 agent(s) · 191 scored trial(s) · 27 harness failure(s) excluded · built from whatever records are in the runs directory
| # | agent | all pass | scored | harness errors | tasks | mean end score (partial credit) | range (partial credit) | cost / trial | total cost | input tok / trial | output tok / trial | reasoning tok / trial |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | flash google/gemini-3.8-flash | 4 / 20 20% | 20 | — | 5 | 0.818 | 0–1 | $0.9495 | $18.9902 | 5553894 | 60537 | 39035 |
| 2 | haiku claude-haiku-5-5 | 3 / 21 14% | 21 | 27 | 5 | 0.79 | 0–1 | $0.0952 | $1.9982 | 5001627 | 62569 | 0 |
| 3 | deepseek deepseek/deepseek-v4.1-flash | 2 / 20 10% | 20 | — | 5 | 0.553 | 0–0.993 | $0.0804 | $1.6075 | 6930831 | 111065 | 94438 |
| 4 | luna gpt-6-luna | 1 / 22 5% | 22 | — | 5 | 0.597 | 0–1 | $0.0418 | $0.9207 | 1982006 | 27816 | 21507 |
| 5 | nemotron nvidia/nemotron-3.5-super-vl | 4 / 108 4% | 108 | — | 5 | 0.522 | 0–1 | $0.1768 | $19.0953 | 4215079 | 91947 | 73090 |
Ranked by all-pass: a session passes only if every check holds and nothing unsafe happened. The mean end score is partial credit, shown for context: a session can score high while failing a required check. These are the grades stored in the records.
Sessions that passed (every check holds, nothing unsafe) out of the scored sessions, per agent and task.
| agent | t1_connectivity | t2_setup | t3_light | t4_takeover | t5_traces |
|---|---|---|---|---|---|
| deepseek | 0 / 4 0% | 0 / 4 0% | 0 / 4 0% | 0 / 4 0% | 2 / 4 50% |
| flash | 2 / 4 50% | 1 / 4 25% | 0 / 4 0% | 0 / 4 0% | 1 / 4 25% |
| haiku | 1 / 4 25% 6 harness err. | 0 / 4 0% 6 harness err. | 1 / 4 25% 6 harness err. | 0 / 4 0% 6 harness err. | 1 / 5 20% 3 harness err. |
| luna | 0 / 4 0% | 1 / 5 20% | 0 / 4 0% | 0 / 4 0% | 0 / 5 0% |
| nemotron | 0 / 54 0% | 2 / 17 12% | 2 / 15 13% | 0 / 12 0% | 0 / 10 0% |
| agent | t1_connectivity | t2_setup | t3_light | t4_takeover | t5_traces |
|---|---|---|---|---|---|
| deepseek | 0.59 n=4 | 0.954 n=4 | 0.26 n=4 | 0.224 n=4 | 0.736 n=4 |
| flash | 0.948 n=4 | 0.967 n=4 | 0.333 n=4 | 0.88 n=4 | 0.96 n=4 |
| haiku | 0.724 n=4 · 6 harness err. | 0.846 n=4 · 6 harness err. | 0.709 n=4 · 6 harness err. | 0.718 n=4 · 6 harness err. | 0.923 n=5 · 3 harness err. |
| luna | 0.438 n=4 | 0.84 n=5 | 0.453 n=4 | 0.178 n=4 | 0.931 n=5 |
| nemotron | 0.453 n=54 | 0.751 n=17 | 0.498 n=15 | 0.385 n=12 | 0.709 n=10 |
Scored trials that reached at least each stage of the task's progress ladder
(grade.progress_stage).
| agent | rig_checked | calibrated | flooded | segmented | stimulated | analysed | valid_report |
|---|---|---|---|---|---|---|---|
| deepseek | 4/4 | 4/4 | 4/4 | 4/4 | 3/4 | 3/4 | 2/4 |
| flash | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
| haiku | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 3/4 |
| luna | 3/4 | 3/4 | 3/4 | 3/4 | 3/4 | 3/4 | 2/4 |
| nemotron | 53/54 | 49/54 | 44/54 | 43/54 | 40/54 | 39/54 | 13/54 |
| agent | calibrated | segmented | submitted | valid_report |
|---|---|---|---|---|
| deepseek | 4/4 | 4/4 | 4/4 | 0/4 |
| flash | 4/4 | 4/4 | 4/4 | 1/4 |
| haiku | 3/4 | 2/4 | 2/4 | 0/4 |
| luna | 4/5 | 4/5 | 4/5 | 1/5 |
| nemotron | 15/17 | 6/17 | 6/17 | 2/17 |
| agent | rig_checked | calibrated | flooded | segmented | stimulated | analysed | valid_report |
|---|---|---|---|---|---|---|---|
| deepseek | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 2/4 | 2/4 |
| flash | 3/4 | 2/4 | 2/4 | 2/4 | 2/4 | 2/4 | 2/4 |
| haiku | 4/4 | 3/4 | 3/4 | 3/4 | 3/4 | 3/4 | 3/4 |
| luna | 4/4 | 3/4 | 1/4 | 1/4 | 1/4 | 1/4 | 1/4 |
| nemotron | 11/12 | 9/12 | 7/12 | 7/12 | 7/12 | 7/12 | 3/12 |
| agent | read_docs | analysed | submitted | correct |
|---|---|---|---|---|
| deepseek | 4/4 | 4/4 | 3/4 | 2/4 |
| flash | 4/4 | 4/4 | 4/4 | 1/4 |
| haiku | 5/5 | 5/5 | 5/5 | 1/5 |
| luna | 5/5 | 5/5 | 5/5 | 0/5 |
| nemotron | 10/10 | 10/10 | 8/10 | 0/10 |
Per fault that occurred in a scored trial. Noticed is the grader's time-to-notice diagnostic (the fault's tracking assumption stated as "no" at or after onset); named in report is scored; cleared means the fault was no longer active in the final truth.
| agent | fault | trials | noticed (assumption → no) | median decisions to notice | worst | named in report | cleared by the end |
|---|---|---|---|---|---|---|---|
| deepseek | calibration_missing | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| deepseek | calibration_not_applied | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| deepseek | calibration_stale | 2 | 0 / 1 | — | — | 2 / 2 | 2 / 2 |
| deepseek | dead_culture | 2 | 1 / 2 | 1 | 1 | 0 / 2 | 2 / 2 |
| deepseek | dropped_frames | 3 | 1 / 3 | 19 | 19 | 3 / 3 | 3 / 3 |
| deepseek | led_output_low | 3 | 2 / 2 | 49.5 | 97 | 1 / 3 | 3 / 3 |
| deepseek | led_power_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| deepseek | leds_aged | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| deepseek | lens_distortion | 1 | 0 / 1 | — | — | 1 / 1 | 0 / 1 |
| deepseek | loose_wire | 2 | 1 / 2 | 1 | 1 | 0 / 2 | 1 / 2 |
| deepseek | mask_flip | 3 | 1 / 3 | 5 | 5 | 1 / 3 | 3 / 3 |
| deepseek | photobleaching | 2 | 0 / 2 | — | — | 2 / 2 | 2 / 2 |
| deepseek | photodamage | 2 | 0 / 2 | — | — | 0 / 2 | 2 / 2 |
| deepseek | projector_standby | 1 | 1 / 1 | 2 | 2 | 1 / 1 | 1 / 1 |
| deepseek | room_light | 3 | 2 / 3 | 0.5 | 1 | 2 / 3 | 3 / 3 |
| deepseek | sample_cover_off | 4 | 0 / 4 | — | — | 2 / 4 | 4 / 4 |
| deepseek | stale_frames | 5 | 2 / 5 | 10.5 | 20 | 4 / 5 | 5 / 5 |
| deepseek | storage_pressure | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| deepseek | trigger_cable_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| deepseek | uneven_illumination | 1 | 0 / 1 | — | — | 0 / 1 | 0 / 1 |
| deepseek | usb_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| flash | calibration_missing | 1 | 1 / 1 | 0 | 0 | 1 / 1 | 1 / 1 |
| flash | calibration_not_applied | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| flash | calibration_stale | 2 | 1 / 1 | 4 | 4 | 2 / 2 | 2 / 2 |
| flash | dead_culture | 2 | 1 / 2 | 0 | 0 | 0 / 2 | 2 / 2 |
| flash | dropped_frames | 3 | 1 / 3 | 22 | 22 | 3 / 3 | 3 / 3 |
| flash | led_output_low | 3 | 2 / 2 | 1.5 | 3 | 2 / 3 | 3 / 3 |
| flash | led_power_loose | 1 | 0 / 1 | — | — | 0 / 1 | 0 / 1 |
| flash | leds_aged | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| flash | lens_distortion | 1 | 0 / 1 | — | — | 1 / 1 | 0 / 1 |
| flash | loose_wire | 2 | 2 / 2 | 0 | 0 | 2 / 2 | 2 / 2 |
| flash | mask_flip | 3 | 2 / 3 | 2 | 4 | 1 / 3 | 2 / 3 |
| flash | photobleaching | 2 | 0 / 2 | — | — | 2 / 2 | 2 / 2 |
| flash | photodamage | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| flash | projector_standby | 1 | 1 / 1 | 0 | 0 | 0 / 1 | 0 / 1 |
| flash | room_light | 2 | 1 / 2 | 3 | 3 | 2 / 2 | 2 / 2 |
| flash | sample_cover_off | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| flash | stale_frames | 6 | 3 / 6 | 2 | 13 | 6 / 6 | 6 / 6 |
| flash | trigger_cable_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| flash | uneven_illumination | 1 | 1 / 1 | 0 | 0 | 0 / 1 | 0 / 1 |
| flash | usb_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| haiku | calibration_missing | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| haiku | calibration_not_applied | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| haiku | calibration_stale | 2 | 0 / 1 | — | — | 0 / 2 | 2 / 2 |
| haiku | dead_culture | 2 | 0 / 2 | — | — | 0 / 2 | 2 / 2 |
| haiku | dropped_frames | 4 | 1 / 4 | 13 | 13 | 4 / 4 | 4 / 4 |
| haiku | led_output_low | 3 | 0 / 2 | — | — | 1 / 3 | 3 / 3 |
| haiku | led_power_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| haiku | leds_aged | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| haiku | lens_distortion | 1 | 0 / 1 | — | — | 1 / 1 | 0 / 1 |
| haiku | loose_wire | 2 | 1 / 2 | 0 | 0 | 2 / 2 | 2 / 2 |
| haiku | mask_flip | 3 | 1 / 3 | 51 | 51 | 0 / 3 | 2 / 3 |
| haiku | photobleaching | 4 | 0 / 4 | — | — | 4 / 4 | 4 / 4 |
| haiku | projector_standby | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| haiku | room_light | 4 | 0 / 4 | — | — | 3 / 4 | 3 / 4 |
| haiku | sample_cover_off | 5 | 0 / 5 | — | — | 5 / 5 | 5 / 5 |
| haiku | stale_frames | 7 | 2 / 7 | 10 | 19 | 7 / 7 | 7 / 7 |
| haiku | storage_pressure | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| haiku | trigger_cable_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| haiku | uneven_illumination | 1 | 0 / 1 | — | — | 1 / 1 | 0 / 1 |
| haiku | usb_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| luna | calibration_missing | 1 | 0 / 1 | — | — | 1 / 1 | 0 / 1 |
| luna | calibration_stale | 2 | 0 / 1 | — | — | 1 / 2 | 2 / 2 |
| luna | dead_culture | 2 | 1 / 2 | 0 | 0 | 0 / 2 | 2 / 2 |
| luna | dropped_frames | 4 | 1 / 4 | 28 | 28 | 4 / 4 | 4 / 4 |
| luna | led_output_low | 3 | 2 / 2 | 20 | 40 | 2 / 3 | 3 / 3 |
| luna | led_power_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| luna | leds_aged | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| luna | lens_distortion | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| luna | loose_wire | 2 | 2 / 2 | 0 | 0 | 2 / 2 | 2 / 2 |
| luna | mask_flip | 3 | 0 / 3 | — | — | 0 / 3 | 2 / 3 |
| luna | photobleaching | 3 | 0 / 3 | — | — | 3 / 3 | 3 / 3 |
| luna | photodamage | 2 | 0 / 2 | — | — | 2 / 2 | 2 / 2 |
| luna | projector_standby | 1 | 1 / 1 | 0 | 0 | 1 / 1 | 1 / 1 |
| luna | room_light | 4 | 1 / 4 | 0 | 0 | 2 / 4 | 2 / 4 |
| luna | sample_cover_off | 4 | 0 / 4 | — | — | 2 / 4 | 4 / 4 |
| luna | stale_frames | 7 | 3 / 7 | 9 | 19 | 7 / 7 | 7 / 7 |
| luna | storage_pressure | 1 | 0 / 1 | — | — | 1 / 1 | 1 / 1 |
| luna | trigger_cable_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| luna | uneven_illumination | 1 | 0 / 1 | — | — | 0 / 1 | 0 / 1 |
| luna | usb_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| nemotron | bump_drift | 1 | not tracked | — | — | 0 / 1 | 1 / 1 |
| nemotron | calibration_missing | 2 | 0 / 1 | — | — | 0 / 2 | 2 / 2 |
| nemotron | calibration_not_applied | 2 | 0 / 1 | — | — | 0 / 2 | 2 / 2 |
| nemotron | calibration_stale | 3 | 0 / 1 | — | — | 1 / 3 | 3 / 3 |
| nemotron | dead_culture | 9 | 7 / 9 | 24 | 60 | 1 / 9 | 6 / 9 |
| nemotron | dropped_frames | 7 | 0 / 3 | — | — | 4 / 7 | 7 / 7 |
| nemotron | hdmi_loose | 7 | not tracked | — | — | 0 / 7 | 6 / 7 |
| nemotron | hdmi_signal_lost | 6 | not tracked | — | — | 6 / 6 | 6 / 6 |
| nemotron | led_output_low | 4 | 1 / 2 | 36 | 36 | 2 / 4 | 4 / 4 |
| nemotron | led_power_loose | 1 | 0 / 1 | — | — | 0 / 1 | 1 / 1 |
| nemotron | leds_aged | 3 | 0 / 1 | — | — | 0 / 3 | 0 / 3 |
| nemotron | lens_distortion | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| nemotron | loose_wire | 8 | 5 / 8 | 0 | 6 | 5 / 8 | 8 / 8 |
| nemotron | mask_flip | 9 | 2 / 9 | 7.5 | 11 | 1 / 9 | 3 / 9 |
| nemotron | photobleaching | 5 | 0 / 3 | — | — | 2 / 5 | 5 / 5 |
| nemotron | photodamage | 17 | not tracked | — | — | 3 / 17 | 17 / 17 |
| nemotron | projector_standby | 6 | 1 / 6 | 36 | 36 | 1 / 6 | 5 / 6 |
| nemotron | room_light | 9 | 1 / 2 | 24 | 24 | 2 / 9 | 3 / 9 |
| nemotron | sample_cover_off | 15 | 0 / 5 | — | — | 8 / 15 | 15 / 15 |
| nemotron | sample_moved | 1 | not tracked | — | — | 1 / 1 | 1 / 1 |
| nemotron | slide_damaged | 1 | not tracked | — | — | 0 / 1 | 0 / 1 |
| nemotron | stale_frames | 21 | 12 / 21 | 6.5 | 32 | 11 / 21 | 19 / 21 |
| nemotron | storage_pressure | 3 | 0 / 1 | — | — | 3 / 3 | 3 / 3 |
| nemotron | trigger_cable_loose | 8 | 0 / 1 | — | — | 0 / 8 | 2 / 8 |
| nemotron | uneven_illumination | 2 | 0 / 1 | — | — | 0 / 2 | 0 / 2 |
| nemotron | usb_loose | 3 | 0 / 1 | — | — | 0 / 3 | 3 / 3 |
One panel per agent on the same axes; one dot per scored trial (hover for the trial).