E5 · Scenario sweep: 65 test scenarios × 1 trial on the frozen code (no change to what the model sees); terra and sol, experiments/sweep1_openai. All-pass: Terra 44/65, Sol 48/65. Not part of the headline.

Compare models

2 agent(s) · 128 scored trial(s) · 2 harness failure(s) excluded · built from whatever records are in the runs directory

Leaderboard

#agent mean end scorerangeall pass scoredharness errorstasks cost / trialtotal cost input tok / trialoutput tok / trial reasoning tok / trial
1terra
gpt-6.1-terra
0.9510–144 / 6565—5$0.1534$9.9713127909489484192
2sol
gpt-6.1-sol
0.9230–148 / 636325$0.2661$17.9984122283273842472

Mean of each trial's end grade over the scored trials. Harness failures are counted in their own column and never scored. Means over a handful of trials are noisy; read the range and the per-task table before ranking.

Mean end score by task

agentt1_connectivityt2_setupt3_lightt4_takeovert5_traces
sol0.84
n=14
0.964
n=15
0.923
n=14 · 2 harness err.
0.909
n=11
0.998
n=9
terra0.984
n=14
0.951
n=15
0.935
n=16
0.905
n=11
0.98
n=9

Progress reached

Scored trials that reached at least each stage of the task's progress ladder (grade.progress_stage).

t1_connectivity

agentrig_checkedcalibratedfloodedsegmentedstimulatedanalysedvalid_report
sol14/1414/1414/1414/1414/1414/1414/14
terra14/1414/1414/1414/1414/1414/1414/14

t2_setup

agentcalibratedsegmentedsubmittedvalid_report
sol15/1515/1515/159/15
terra14/1513/1513/158/15

t4_takeover

agentrig_checkedcalibratedfloodedsegmentedstimulatedanalysedvalid_report
sol11/1111/1111/1111/1111/1111/1111/11
terra11/1111/1111/1111/1111/1111/1111/11

t5_traces

agentread_docsanalysedsubmittedcorrect
sol9/99/99/99/9
terra9/99/99/98/9

Recovery by fault

Per fault that occurred in a scored trial. Noticed is the grader's time-to-notice diagnostic (the fault's tracking assumption stated as "no" at or after onset); named in report is scored; cleared means the fault was no longer active in the final truth.

agentfaulttrialsnoticed (assumption → no)median decisions to noticeworstnamed in reportcleared by the end
solbump_drift11 / 1331 / 11 / 1
solcalibration_missing22 / 2002 / 22 / 2
solcalibration_not_applied20 / 2——0 / 22 / 2
solcalibration_stale10 / 1——0 / 11 / 1
soldead_culture44 / 413252 / 44 / 4
soldropped_frames52 / 58165 / 55 / 5
solhdmi_loose30 / 3——0 / 33 / 3
solhdmi_signal_lost33 / 31143 / 33 / 3
solled_output_low62 / 6005 / 66 / 6
solled_power_loose20 / 2——0 / 22 / 2
solleds_aged20 / 2——0 / 20 / 2
sollens_distortion21 / 2331 / 20 / 2
solloose_wire55 / 5045 / 55 / 5
solmask_flip33 / 3233 / 33 / 3
solphotobleaching40 / 4——4 / 44 / 4
solprojector_standby32 / 32.553 / 33 / 3
solred_led_aged20 / 2——0 / 20 / 2
solroom_light84 / 8127 / 88 / 8
solsample_cover_off40 / 4——4 / 44 / 4
solsample_moved20 / 2——2 / 22 / 2
solslide_damaged11 / 1331 / 10 / 1
solstale_frames95 / 91159 / 99 / 9
solstorage_pressure20 / 2——2 / 22 / 2
soltrigger_cable_loose20 / 2——0 / 22 / 2
soluneven_illumination22 / 26.5132 / 20 / 2
solusb_loose20 / 2——0 / 22 / 2
terrabump_drift11 / 1440 / 11 / 1
terracalibration_missing21 / 2002 / 22 / 2
terracalibration_not_applied20 / 2——0 / 22 / 2
terracalibration_stale10 / 1——1 / 11 / 1
terradead_culture44 / 49192 / 44 / 4
terradropped_frames52 / 56125 / 55 / 5
terrahdmi_loose30 / 3——0 / 33 / 3
terrahdmi_signal_lost33 / 31133 / 33 / 3
terraled_output_low65 / 67306 / 66 / 6
terraled_power_loose20 / 2——0 / 22 / 2
terraleds_aged20 / 2——0 / 20 / 2
terralens_distortion20 / 2——0 / 20 / 2
terraloose_wire55 / 5045 / 55 / 5
terramask_flip33 / 3053 / 33 / 3
terraphotobleaching30 / 3——3 / 33 / 3
terraprojector_standby31 / 3002 / 33 / 3
terrared_led_aged20 / 2——0 / 20 / 2
terraroom_light84 / 80.527 / 87 / 8
terrasample_cover_off40 / 4——4 / 44 / 4
terrasample_moved20 / 2——2 / 22 / 2
terraslide_damaged11 / 1441 / 10 / 1
terrastale_frames106 / 1001310 / 1010 / 10
terrastorage_pressure10 / 1——1 / 11 / 1
terratrigger_cable_loose20 / 2——0 / 22 / 2
terrauneven_illumination21 / 218182 / 20 / 2
terrausb_loose20 / 2——0 / 22 / 2

Cost against end score

One panel per agent on the same axes; one dot per scored trial (hover for the trial).

sol 63 scored trial(s)
00.510$0.341$0.681t1_connectivity · dead_culture · trial 0: end score 1, $0.3969, 11875 output tokenst1_connectivity · hdmi_loose · trial 0: end score 1, $0.2517, 7386 output tokenst1_connectivity · led_power_loose · trial 0: end score 0.846, $0.2420, 6570 output tokenst1_connectivity · loose_wire · trial 0: end score 1, $0.3825, 9592 output tokenst1_connectivity · low_disk · trial 0: end score 0, $0.1618, 4141 output tokenst1_connectivity · mask_flip · trial 0: end score 1, $0.3878, 9654 output tokenst1_connectivity · none · trial 0: end score 1, $0.3382, 7782 output tokenst1_connectivity · operator_away · trial 0: end score 1, $0.2920, 6840 output tokenst1_connectivity · projector_standby · trial 0: end score 1, $0.3206, 11075 output tokenst1_connectivity · red_led_aged · trial 0: end score 0.957, $0.4063, 10685 output tokenst1_connectivity · room_light · trial 0: end score 0, $0.3612, 7874 output tokenst1_connectivity · stale_frames · trial 0: end score 1, $0.2756, 7544 output tokenst1_connectivity · trigger_cable · trial 0: end score 0.96, $0.3068, 10496 output tokenst1_connectivity · usb_loose · trial 0: end score 1, $0.2945, 8223 output tokenst2_setup · calibration_missing · trial 0: end score 0.956, $0.0893, 2325 output tokenst2_setup · calibration_stale · trial 0: end score 0.867, $0.0924, 2220 output tokenst2_setup · dense_culture · trial 0: end score 1, $0.0973, 2526 output tokenst2_setup · distortion_dense · trial 0: end score 0.867, $0.0921, 2048 output tokenst2_setup · few_markers · trial 0: end score 1, $0.2191, 4032 output tokenst2_setup · hdmi_loose · trial 0: end score 1, $0.0707, 1428 output tokenst2_setup · lens_distortion · trial 0: end score 1, $0.1111, 3562 output tokenst2_setup · loose_wire · trial 0: end score 1, $0.1010, 2178 output tokenst2_setup · mask_flip · trial 0: end score 0.956, $0.0899, 2637 output tokenst2_setup · missing_uneven · trial 0: end score 0.956, $0.1520, 4166 output tokenst2_setup · none · trial 0: end score 1, $0.0775, 1352 output tokenst2_setup · operator_away · trial 0: end score 1, $0.0918, 1943 output tokenst2_setup · projector_standby · trial 0: end score 1, $0.1194, 2597 output tokenst2_setup · room_light · trial 0: end score 0.867, $0.0879, 2076 output tokenst2_setup · uneven_illumination · trial 0: end score 1, $0.1224, 3581 output tokenst3_light · close_neighbours · trial 0: end score 0.977, $0.3030, 7269 output tokenst3_light · cluster_fast_bleach · trial 0: end score 0.986, $0.3095, 7665 output tokenst3_light · converging_led_low · trial 0: end score 0.973, $0.5479, 10891 output tokenst3_light · crosstalk_two_targets · trial 0: end score 0.828, $0.2775, 7782 output tokenst3_light · dead_culture · trial 0: end score 0.976, $0.3630, 7199 output tokenst3_light · fast_bleach · trial 0: end score 0.667, $0.3051, 7845 output tokenst3_light · led_low · trial 0: end score 0.981, $0.3477, 8999 output tokenst3_light · loose_wire · trial 0: end score 0.968, $0.3380, 8624 output tokenst3_light · none · trial 0: end score 1, $0.2488, 5671 output tokenst3_light · red_led_aged · trial 0: end score 0.986, $0.3734, 8220 output tokenst3_light · room_light · trial 0: end score 0.667, $0.3872, 7893 output tokenst3_light · strong_crosstalk · trial 0: end score 0.992, $0.2706, 7430 output tokenst3_light · strong_input · trial 0: end score 0.94, $0.3044, 7543 output tokenst3_light · two_targets_room_light · trial 0: end score 0.98, $0.6487, 16187 output tokenst4_takeover · clean · trial 0: end score 1, $0.2703, 6928 output tokenst4_takeover · dead_culture_swap · trial 0: end score 1, $0.3158, 8523 output tokenst4_takeover · disk_note_fine · trial 0: end score 0, $0.2304, 6313 output tokenst4_takeover · disk_triage · trial 0: end score 1, $0.2437, 6848 output tokenst4_takeover · flip_grid_check · trial 0: end score 1, $0.3729, 9636 output tokenst4_takeover · frozen_counters · trial 0: end score 1, $0.2778, 7629 output tokenst4_takeover · loose_jumper · trial 0: end score 1, $0.2124, 6742 output tokenst4_takeover · moved_sample · trial 0: end score 1, $0.3245, 7633 output tokenst4_takeover · t_dead_jumper · trial 0: end score 1, $0.2639, 7101 output tokenst4_takeover · t_moved_frozen · trial 0: end score 1, $0.3121, 8156 output tokenst4_takeover · t_present_faults · trial 0: end score 1, $0.3856, 10390 output tokenst5_traces · bleached · trial 0: end score 0.999, $0.3213, 13526 output tokenst5_traces · bleached_and_dropped · trial 0: end score 0.998, $0.3091, 10619 output tokenst5_traces · clean · trial 0: end score 0.998, $0.2602, 10582 output tokenst5_traces · dropped_and_light · trial 0: end score 0.997, $0.2590, 11695 output tokenst5_traces · everything · trial 0: end score 0.997, $0.3250, 11966 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.999, $0.2714, 11820 output tokenst5_traces · frozen_stretch · trial 0: end score 1, $0.2783, 12412 output tokenst5_traces · light_and_frozen · trial 0: end score 0.999, $0.2715, 11356 output tokenst5_traces · off_target · trial 0: end score 0.998, $0.2010, 7671 output tokenscost per trial (USD)
terra 65 scored trial(s)
00.510$0.341$0.681t1_connectivity · dead_culture · trial 0: end score 0.883, $0.2102, 9431 output tokenst1_connectivity · hdmi_loose · trial 0: end score 1, $0.1700, 10248 output tokenst1_connectivity · led_power_loose · trial 0: end score 1, $0.1231, 7631 output tokenst1_connectivity · loose_wire · trial 0: end score 1, $0.1930, 11737 output tokenst1_connectivity · low_disk · trial 0: end score 1, $0.1313, 7908 output tokenst1_connectivity · mask_flip · trial 0: end score 0.969, $0.1717, 9486 output tokenst1_connectivity · none · trial 0: end score 1, $0.1262, 7638 output tokenst1_connectivity · operator_away · trial 0: end score 1, $0.1276, 9801 output tokenst1_connectivity · projector_standby · trial 0: end score 1, $0.1340, 7910 output tokenst1_connectivity · red_led_aged · trial 0: end score 0.969, $0.1706, 11610 output tokenst1_connectivity · room_light · trial 0: end score 1, $0.2241, 8611 output tokenst1_connectivity · stale_frames · trial 0: end score 1, $0.2215, 12225 output tokenst1_connectivity · trigger_cable · trial 0: end score 0.96, $0.1329, 8308 output tokenst1_connectivity · usb_loose · trial 0: end score 1, $0.1543, 9471 output tokenst2_setup · calibration_missing · trial 0: end score 0.956, $0.0451, 2527 output tokenst2_setup · calibration_stale · trial 0: end score 1, $0.0666, 4040 output tokenst2_setup · dense_culture · trial 0: end score 1, $0.0852, 3894 output tokenst2_setup · distortion_dense · trial 0: end score 0.867, $0.0697, 3330 output tokenst2_setup · few_markers · trial 0: end score 0.956, $0.1263, 6618 output tokenst2_setup · hdmi_loose · trial 0: end score 1, $0.0489, 2320 output tokenst2_setup · lens_distortion · trial 0: end score 0.867, $0.0538, 2832 output tokenst2_setup · loose_wire · trial 0: end score 1, $0.0604, 3075 output tokenst2_setup · mask_flip · trial 0: end score 1, $0.0625, 2985 output tokenst2_setup · missing_uneven · trial 0: end score 0.948, $0.1132, 7789 output tokenst2_setup · none · trial 0: end score 1, $0.0566, 2108 output tokenst2_setup · operator_away · trial 0: end score 1, $0.0509, 2003 output tokenst2_setup · projector_standby · trial 0: end score 1, $0.0613, 2383 output tokenst2_setup · room_light · trial 0: end score 0.867, $0.0398, 1944 output tokenst2_setup · uneven_illumination · trial 0: end score 0.811, $0.0989, 6021 output tokenst3_light · close_neighbours · trial 0: end score 0.998, $0.1269, 6050 output tokenst3_light · cluster_fast_bleach · trial 0: end score 0.998, $0.1630, 8994 output tokenst3_light · converging · trial 0: end score 0.943, $0.3581, 15013 output tokenst3_light · converging_led_low · trial 0: end score 0.667, $0.3755, 14400 output tokenst3_light · crosstalk_two_targets · trial 0: end score 0.72, $0.1712, 8460 output tokenst3_light · dead_culture · trial 0: end score 0.975, $0.1410, 6103 output tokenst3_light · fast_bleach · trial 0: end score 0.991, $0.1306, 7003 output tokenst3_light · led_low · trial 0: end score 0.898, $0.1619, 9207 output tokenst3_light · loose_wire · trial 0: end score 0.984, $0.1451, 7222 output tokenst3_light · none · trial 0: end score 0.981, $0.1294, 5497 output tokenst3_light · red_led_aged · trial 0: end score 0.959, $0.1747, 9394 output tokenst3_light · room_light · trial 0: end score 0.967, $0.1827, 9034 output tokenst3_light · stale_frames · trial 0: end score 0.969, $0.2020, 10966 output tokenst3_light · strong_crosstalk · trial 0: end score 0.976, $0.1853, 10310 output tokenst3_light · strong_input · trial 0: end score 0.953, $0.1473, 7810 output tokenst3_light · two_targets_room_light · trial 0: end score 0.977, $0.2567, 13294 output tokenst4_takeover · clean · trial 0: end score 1, $0.1334, 7790 output tokenst4_takeover · dead_culture_swap · trial 0: end score 1, $0.3295, 15783 output tokenst4_takeover · disk_note_fine · trial 0: end score 1, $0.1249, 8457 output tokenst4_takeover · disk_triage · trial 0: end score 0.998, $0.1662, 9579 output tokenst4_takeover · flip_grid_check · trial 0: end score 1, $0.1806, 10488 output tokenst4_takeover · frozen_counters · trial 0: end score 1, $0.1466, 8932 output tokenst4_takeover · loose_jumper · trial 0: end score 0.971, $0.1365, 7431 output tokenst4_takeover · moved_sample · trial 0: end score 1, $0.1714, 9071 output tokenst4_takeover · t_dead_jumper · trial 0: end score 0.988, $0.1551, 10748 output tokenst4_takeover · t_moved_frozen · trial 0: end score 0.996, $0.2854, 16409 output tokenst4_takeover · t_present_faults · trial 0: end score 0, $0.2240, 11473 output tokenst5_traces · bleached · trial 0: end score 0.847, $0.1739, 13092 output tokenst5_traces · bleached_and_dropped · trial 0: end score 0.993, $0.2199, 17945 output tokenst5_traces · clean · trial 0: end score 0.999, $0.1496, 12361 output tokenst5_traces · dropped_and_light · trial 0: end score 0.995, $0.1782, 14128 output tokenst5_traces · everything · trial 0: end score 0.998, $0.2003, 19420 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.997, $0.1803, 15076 output tokenst5_traces · frozen_stretch · trial 0: end score 0.999, $0.1813, 15850 output tokenst5_traces · light_and_frozen · trial 0: end score 0.999, $0.1242, 12672 output tokenst5_traces · off_target · trial 0: end score 0.997, $0.1286, 10272 output tokenscost per trial (USD)