Compare models

13 agent(s) · 195 scored trial(s) · built from whatever records are in the runs directory

Leaderboard

#agent all pass scoredharness errorstasks mean end score (partial credit)range (partial credit) cost / trialtotal cost input tok / trialoutput tok / trial reasoning tok / trial
1fable
claude-fable-5-1
9 / 15 60%15—50.9570.791–1$5.6905$85.35694391092529900
2terra
gpt-6.1-terra
9 / 15 60%15—50.9210–1$0.1972$2.95741529886116105588
3opus
claude-opus-5-5
8 / 15 53%15—50.9580.807–1$1.8345$27.51743823620293880
4sol
gpt-6.1-sol
8 / 15 53%15—50.9010–1$0.3369$5.0533158228191382994
5astra
gpt-6-astra
8 / 15 53%15—50.7860–1$2.3337$35.0056149766782262540
6sonnet
claude-sonnet-5-5
7 / 15 47%15—50.9660.818–1$1.1999$17.99814915882416370
7argon
google/gemini-4-argon
7 / 15 47%15—50.9450.667–1$1.9424$29.136757322125573526197
8flash
google/gemini-3.8-flash
3 / 15 20%15—50.6470–1$0.9165$13.747755234205974440689
9deepseek
deepseek/deepseek-v4.1-flash
3 / 15 20%15—50.480–0.997$0.0816$1.2246812499211044292096
10haiku
claude-haiku-5-5
1 / 15 7%15—50.6860–0.981$0.2983$4.47385906308692310
11gemini_pro
google/gemini-3.1-pro-preview
1 / 15 7%15—50.4270–0.979$1.4421$21.630824658772295311923
12luna
gpt-6-luna
0 / 15 0%15—50.3190–0.922$0.0379$0.568619438412411418779
13nemotron
nvidia/nemotron-3.5-super-vl
0 / 15 0%15—50.3180–0.974$0.2092$3.1382619457211718299456

Ranked by all-pass: a session passes only if every check holds and nothing unsafe happened. The mean end score is partial credit, shown for context: a session can score high while failing a required check. These are the grades stored in the records: the original grade, which is the headline. The audit's alternative interpretation of the documentation is shown beside it on Results.

All pass by task

Sessions that passed (every check holds, nothing unsafe) out of the scored sessions, per agent and task.

agentt1_connectivityt2_setupt3_lightt4_takeovert5_traces
argon1 / 3 33%2 / 3 67%1 / 3 33%0 / 3 0%3 / 3 100%
astra2 / 3 67%0 / 3 0%1 / 3 33%2 / 3 67%3 / 3 100%
deepseek0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%3 / 3 100%
fable1 / 3 33%3 / 3 100%2 / 3 67%0 / 3 0%3 / 3 100%
flash1 / 3 33%0 / 3 0%0 / 3 0%0 / 3 0%2 / 3 67%
gemini_pro0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%1 / 3 33%
haiku0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%1 / 3 33%
luna0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%
nemotron0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%0 / 3 0%
opus1 / 3 33%3 / 3 100%1 / 3 33%0 / 3 0%3 / 3 100%
sol3 / 3 100%1 / 3 33%1 / 3 33%0 / 3 0%3 / 3 100%
sonnet3 / 3 100%2 / 3 67%0 / 3 0%0 / 3 0%2 / 3 67%
terra3 / 3 100%0 / 3 0%1 / 3 33%2 / 3 67%3 / 3 100%

Mean end score by task (partial credit)

agentt1_connectivityt2_setupt3_lightt4_takeovert5_traces
argon0.956
n=3
0.991
n=3
0.831
n=3
0.946
n=3
0.999
n=3
astra0.986
n=3
0.96
n=3
0.319
n=3
0.667
n=3
0.999
n=3
deepseek0.144
n=3
0.941
n=3
0
n=3
0.319
n=3
0.996
n=3
fable0.915
n=3
1
n=3
0.912
n=3
0.97
n=3
0.992
n=3
flash0.97
n=3
0.85
n=3
0.434
n=3
0
n=3
0.979
n=3
gemini_pro0.559
n=3
0.711
n=3
0
n=3
0
n=3
0.865
n=3
haiku0.402
n=3
0.834
n=3
0.768
n=3
0.516
n=3
0.911
n=3
luna0.246
n=3
0.541
n=3
0
n=3
0
n=3
0.808
n=3
nemotron0
n=3
0.675
n=3
0.095
n=3
0
n=3
0.822
n=3
opus0.945
n=3
1
n=3
0.866
n=3
0.981
n=3
0.998
n=3
sol1
n=3
0.976
n=3
0.879
n=3
0.651
n=3
0.999
n=3
sonnet1
n=3
0.975
n=3
0.887
n=3
0.981
n=3
0.987
n=3
terra1
n=3
0.97
n=3
0.643
n=3
0.994
n=3
0.998
n=3

Progress reached

Scored trials that reached at least each stage of the task's progress ladder (grade.progress_stage).

t1_connectivity

agentrig_checkedcalibratedfloodedsegmentedstimulatedanalysedvalid_report
argon3/33/33/33/33/33/33/3
astra3/33/33/33/33/33/33/3
deepseek3/33/33/33/33/33/31/3
fable3/33/33/33/33/33/33/3
flash3/33/33/33/33/33/33/3
gemini_pro3/33/33/33/33/33/32/3
haiku3/33/33/33/33/33/30/3
luna3/33/33/33/33/33/32/3
nemotron3/33/33/33/33/33/30/3
opus3/33/33/33/33/33/33/3
sol3/33/33/33/33/33/33/3
sonnet3/33/33/33/33/33/33/3
terra3/33/33/33/33/33/33/3

t2_setup

agentcalibratedsegmentedsubmittedvalid_report
argon3/33/33/32/3
astra3/33/33/30/3
deepseek3/32/32/30/3
fable3/33/33/33/3
flash2/31/31/30/3
gemini_pro2/30/30/30/3
haiku2/31/31/30/3
luna0/30/30/30/3
nemotron2/30/30/30/3
opus3/33/33/33/3
sol3/33/33/31/3
sonnet3/32/32/32/3
terra3/33/33/30/3

t4_takeover

agentrig_checkedcalibratedfloodedsegmentedstimulatedanalysedvalid_report
argon3/33/33/33/33/33/33/3
astra3/33/33/33/33/33/33/3
deepseek3/33/33/33/33/32/32/3
fable3/33/33/33/33/33/33/3
flash2/32/32/32/32/32/32/3
gemini_pro3/33/33/33/33/33/31/3
haiku3/33/33/33/33/33/32/3
luna3/33/33/33/33/33/33/3
nemotron1/31/31/31/31/31/31/3
opus3/33/33/33/33/33/33/3
sol2/32/32/32/32/32/32/3
sonnet3/33/33/33/33/33/33/3
terra3/33/33/33/33/33/33/3

t5_traces

agentread_docsanalysedsubmittedcorrect
argon3/33/33/33/3
astra3/33/33/33/3
deepseek3/33/33/33/3
fable3/33/33/33/3
flash3/33/33/32/3
gemini_pro3/33/33/31/3
haiku3/33/33/31/3
luna3/33/33/30/3
nemotron3/33/33/30/3
opus3/33/33/33/3
sol3/33/33/33/3
sonnet3/33/33/32/3
terra3/33/33/33/3

Recovery by fault

Per fault that occurred in a scored trial. Noticed is the grader's time-to-notice diagnostic (the fault's tracking assumption stated as "no" at or after onset); named in report is scored; cleared means the fault was no longer active in the final truth.

agentfaulttrialsnoticed (assumption → no)median decisions to noticeworstnamed in reportcleared by the end
argoncalibration_missing33 / 3003 / 33 / 3
argoncalibration_not_applied30 / 3——0 / 33 / 3
argondropped_frames33 / 311203 / 33 / 3
argonled_output_low33 / 3000 / 33 / 3
argonled_power_loose30 / 3——0 / 33 / 3
argonprojector_standby33 / 3002 / 33 / 3
argonroom_light55 / 5025 / 55 / 5
argonsample_cover_off50 / 5——5 / 55 / 5
argonstale_frames30 / 3——3 / 33 / 3
argonuneven_illumination33 / 3003 / 30 / 3
argonusb_loose30 / 3——0 / 33 / 3
astracalibration_missing33 / 3003 / 33 / 3
astracalibration_not_applied30 / 3——0 / 33 / 3
astradropped_frames33 / 317223 / 33 / 3
astraled_output_low33 / 3003 / 33 / 3
astraled_power_loose30 / 3——0 / 33 / 3
astraphotodamage10 / 1——1 / 11 / 1
astraprojector_standby33 / 3003 / 33 / 3
astraroom_light66 / 60.526 / 66 / 6
astrasample_cover_off60 / 6——6 / 66 / 6
astrastale_frames30 / 3——3 / 33 / 3
astrauneven_illumination33 / 3002 / 30 / 3
astrausb_loose30 / 3——0 / 33 / 3
deepseekcalibration_missing30 / 3——3 / 33 / 3
deepseekcalibration_not_applied30 / 3——0 / 33 / 3
deepseekdropped_frames32 / 337.5383 / 33 / 3
deepseekled_output_low33 / 3000 / 33 / 3
deepseekled_power_loose30 / 3——0 / 33 / 3
deepseekphotodamage50 / 5——2 / 55 / 5
deepseekprojector_standby33 / 3001 / 33 / 3
deepseekroom_light55 / 510424 / 54 / 5
deepseeksample_cover_off60 / 6——3 / 66 / 6
deepseekstale_frames30 / 3——3 / 33 / 3
deepseekuneven_illumination30 / 3——1 / 30 / 3
deepseekusb_loose30 / 3——0 / 32 / 3
fablecalibration_missing30 / 3——3 / 33 / 3
fablecalibration_not_applied30 / 3——0 / 33 / 3
fabledropped_frames33 / 311143 / 33 / 3
fableled_output_low33 / 3000 / 33 / 3
fableled_power_loose30 / 3——0 / 33 / 3
fableprojector_standby33 / 3003 / 33 / 3
fableroom_light66 / 60.516 / 66 / 6
fablesample_cover_off60 / 6——6 / 66 / 6
fablestale_frames30 / 3——3 / 33 / 3
fableuneven_illumination30 / 3——3 / 30 / 3
fableusb_loose30 / 3——0 / 33 / 3
flashcalibration_missing33 / 3053 / 33 / 3
flashcalibration_not_applied30 / 3——0 / 33 / 3
flashdropped_frames33 / 323253 / 33 / 3
flashled_output_low33 / 3000 / 33 / 3
flashled_power_loose30 / 3——0 / 33 / 3
flashprojector_standby33 / 3001 / 32 / 3
flashroom_light54 / 5443 / 52 / 5
flashsample_cover_off50 / 5——4 / 55 / 5
flashstale_frames30 / 3——3 / 33 / 3
flashuneven_illumination33 / 3051 / 30 / 3
flashusb_loose30 / 3——0 / 33 / 3
gemini_procalibration_missing30 / 3——2 / 33 / 3
gemini_procalibration_not_applied30 / 3——0 / 33 / 3
gemini_prodropped_frames33 / 35142 / 33 / 3
gemini_proled_output_low32 / 31.520 / 33 / 3
gemini_proled_power_loose30 / 3——0 / 33 / 3
gemini_prophotobleaching10 / 1——1 / 11 / 1
gemini_prophotodamage40 / 4——1 / 44 / 4
gemini_proprojector_standby32 / 31.521 / 33 / 3
gemini_proroom_light63 / 66202 / 62 / 6
gemini_prosample_cover_off60 / 6——2 / 66 / 6
gemini_prostale_frames30 / 3——3 / 33 / 3
gemini_prouneven_illumination30 / 3——1 / 30 / 3
gemini_prousb_loose30 / 3——0 / 32 / 3
haikucalibration_missing30 / 3——3 / 33 / 3
haikucalibration_not_applied30 / 3——0 / 33 / 3
haikudropped_frames31 / 318182 / 33 / 3
haikuled_output_low32 / 324.5370 / 33 / 3
haikuled_power_loose30 / 3——0 / 33 / 3
haikuprojector_standby32 / 324.5373 / 33 / 3
haikuroom_light53 / 52294 / 51 / 5
haikusample_cover_off70 / 7——6 / 77 / 7
haikustale_frames30 / 3——3 / 33 / 3
haikuuneven_illumination30 / 3——1 / 30 / 3
haikuusb_loose30 / 3——0 / 31 / 3
lunacalibration_missing31 / 324243 / 30 / 3
lunadropped_frames32 / 318.5213 / 33 / 3
lunaled_output_low33 / 3002 / 33 / 3
lunaled_power_loose30 / 3——0 / 33 / 3
lunaphotodamage10 / 1——1 / 11 / 1
lunaprojector_standby33 / 3001 / 33 / 3
lunaroom_light64 / 61210 / 61 / 6
lunasample_cover_off80 / 8——3 / 88 / 8
lunastale_frames30 / 3——3 / 33 / 3
lunauneven_illumination31 / 324240 / 30 / 3
lunausb_loose30 / 3——0 / 33 / 3
nemotroncalibration_missing30 / 3——0 / 32 / 3
nemotroncalibration_not_applied20 / 2——0 / 22 / 2
nemotrondropped_frames30 / 3——3 / 33 / 3
nemotronled_output_low31 / 3662 / 33 / 3
nemotronled_power_loose30 / 3——0 / 32 / 3
nemotronphotodamage10 / 1——0 / 11 / 1
nemotronprojector_standby31 / 3660 / 31 / 3
nemotronroom_light31 / 3220 / 30 / 3
nemotronsample_cover_off80 / 8——6 / 88 / 8
nemotronstale_frames30 / 3——3 / 33 / 3
nemotronuneven_illumination30 / 3——0 / 30 / 3
nemotronusb_loose30 / 3——0 / 30 / 3
opuscalibration_missing33 / 311113 / 33 / 3
opuscalibration_not_applied30 / 3——0 / 33 / 3
opusdropped_frames33 / 317173 / 33 / 3
opusled_output_low33 / 3000 / 33 / 3
opusled_power_loose30 / 3——0 / 33 / 3
opusprojector_standby33 / 3003 / 33 / 3
opusroom_light66 / 60.546 / 66 / 6
opussample_cover_off60 / 6——6 / 66 / 6
opusstale_frames30 / 3——3 / 33 / 3
opusuneven_illumination33 / 311113 / 30 / 3
opususb_loose30 / 3——0 / 33 / 3
solcalibration_missing32 / 310153 / 33 / 3
solcalibration_not_applied30 / 3——0 / 33 / 3
soldropped_frames33 / 313143 / 33 / 3
solled_output_low33 / 3001 / 33 / 3
solled_power_loose30 / 3——0 / 33 / 3
solprojector_standby33 / 3003 / 33 / 3
solroom_light66 / 6126 / 66 / 6
solsample_cover_off60 / 6——6 / 66 / 6
solstale_frames30 / 3——3 / 33 / 3
soluneven_illumination32 / 310153 / 30 / 3
solusb_loose30 / 3——0 / 33 / 3
sonnetcalibration_missing32 / 38.5173 / 33 / 3
sonnetcalibration_not_applied30 / 3——0 / 33 / 3
sonnetdropped_frames33 / 322253 / 33 / 3
sonnetled_output_low33 / 3000 / 33 / 3
sonnetled_power_loose30 / 3——0 / 33 / 3
sonnetprojector_standby33 / 3003 / 33 / 3
sonnetroom_light66 / 6126 / 65 / 6
sonnetsample_cover_off60 / 6——6 / 66 / 6
sonnetstale_frames30 / 3——3 / 33 / 3
sonnetuneven_illumination32 / 38.5173 / 30 / 3
sonnetusb_loose30 / 3——0 / 33 / 3
terracalibration_missing32 / 3363 / 33 / 3
terracalibration_not_applied30 / 3——0 / 33 / 3
terradropped_frames33 / 313143 / 33 / 3
terraled_output_low33 / 3003 / 33 / 3
terraled_power_loose30 / 3——0 / 33 / 3
terraprojector_standby33 / 3002 / 33 / 3
terraroom_light65 / 6126 / 66 / 6
terrasample_cover_off60 / 6——6 / 66 / 6
terrastale_frames30 / 3——3 / 33 / 3
terrauneven_illumination32 / 3363 / 30 / 3
terrausb_loose30 / 3——0 / 33 / 3

Cost against end score

One panel per agent on the same axes; one dot per scored trial (hover for the trial).

argon 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 1, $1.7631, 74866 output tokenst1_connectivity · usb_loose · trial 1: end score 0.917, $2.0416, 25998 output tokenst1_connectivity · usb_loose · trial 2: end score 0.952, $1.7280, 68924 output tokenst2_setup · missing_uneven · trial 0: end score 1, $0.3430, 8024 output tokenst2_setup · missing_uneven · trial 1: end score 0.973, $0.3797, 7912 output tokenst2_setup · missing_uneven · trial 2: end score 1, $0.2919, 7409 output tokenst3_light · two_targets_room_light · trial 0: end score 1, $2.6245, 37532 output tokenst3_light · two_targets_room_light · trial 1: end score 0.828, $3.4773, 108145 output tokenst3_light · two_targets_room_light · trial 2: end score 0.667, $2.5090, 30721 output tokenst4_takeover · t_present_faults · trial 0: end score 0.927, $1.7729, 28522 output tokenst4_takeover · t_present_faults · trial 1: end score 0.956, $2.4672, 77657 output tokenst4_takeover · t_present_faults · trial 2: end score 0.955, $2.2763, 65928 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.996, $1.5369, 47023 output tokenst5_traces · frozen_and_off_target · trial 1: end score 1, $3.6185, 139656 output tokenst5_traces · frozen_and_off_target · trial 2: end score 1, $2.3068, 107706 output tokenscost per trial (USD)
astra 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 1, $2.2998, 7186 output tokenst1_connectivity · usb_loose · trial 1: end score 1, $2.3290, 7456 output tokenst1_connectivity · usb_loose · trial 2: end score 0.957, $2.1267, 8002 output tokenst2_setup · missing_uneven · trial 0: end score 0.973, $0.9051, 3121 output tokenst2_setup · missing_uneven · trial 1: end score 0.933, $1.0786, 4196 output tokenst2_setup · missing_uneven · trial 2: end score 0.973, $0.7975, 3501 output tokenst3_light · two_targets_room_light · trial 0: end score 0, $5.2746, 14290 output tokenst3_light · two_targets_room_light · trial 1: end score 0, $1.5741, 4966 output tokenst3_light · two_targets_room_light · trial 2: end score 0.956, $4.9731, 13101 output tokenst4_takeover · t_present_faults · trial 0: end score 1, $3.0083, 8391 output tokenst4_takeover · t_present_faults · trial 1: end score 0, $1.6005, 5773 output tokenst4_takeover · t_present_faults · trial 2: end score 1, $3.6599, 8921 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.999, $1.4137, 10185 output tokenst5_traces · frozen_and_off_target · trial 1: end score 1, $2.5409, 15073 output tokenst5_traces · frozen_and_off_target · trial 2: end score 1, $1.4236, 9235 output tokenscost per trial (USD)
deepseek 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0.432, $0.0767, 87442 output tokenst1_connectivity · usb_loose · trial 1: end score 0, $0.1010, 170212 output tokenst1_connectivity · usb_loose · trial 2: end score 0, $0.0974, 138953 output tokenst2_setup · missing_uneven · trial 0: end score 0.933, $0.0353, 50869 output tokenst2_setup · missing_uneven · trial 1: end score 0.917, $0.0439, 79844 output tokenst2_setup · missing_uneven · trial 2: end score 0.973, $0.0508, 88386 output tokenst3_light · two_targets_room_light · trial 0: end score 0, $0.1154, 137797 output tokenst3_light · two_targets_room_light · trial 1: end score 0, $0.0689, 116107 output tokenst3_light · two_targets_room_light · trial 2: end score 0, $0.1003, 112798 output tokenst4_takeover · t_present_faults · trial 0: end score 0, $0.1307, 125535 output tokenst4_takeover · t_present_faults · trial 1: end score 0, $0.0902, 121150 output tokenst4_takeover · t_present_faults · trial 2: end score 0.956, $0.0951, 118908 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.995, $0.0782, 102470 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.996, $0.0682, 99534 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.997, $0.0725, 106631 output tokenscost per trial (USD)
fable 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0.791, $5.9012, 55531 output tokenst1_connectivity · usb_loose · trial 1: end score 0.952, $2.8121, 29477 output tokenst1_connectivity · usb_loose · trial 2: end score 1, $5.7601, 50739 output tokenst2_setup · missing_uneven · trial 0: end score 1, $2.5024, 20041 output tokenst2_setup · missing_uneven · trial 1: end score 1, $2.2119, 15086 output tokenst2_setup · missing_uneven · trial 2: end score 1, $2.2961, 15540 output tokenst3_light · two_targets_room_light · trial 0: end score 0.802, $10.7740, 89592 output tokenst3_light · two_targets_room_light · trial 1: end score 0.962, $7.5294, 78233 output tokenst3_light · two_targets_room_light · trial 2: end score 0.972, $6.4541, 68135 output tokenst4_takeover · t_present_faults · trial 0: end score 0.953, $9.4815, 77905 output tokenst4_takeover · t_present_faults · trial 1: end score 0.981, $7.1882, 56508 output tokenst4_takeover · t_present_faults · trial 2: end score 0.974, $7.5607, 62042 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.999, $4.8039, 61557 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.995, $5.4050, 59533 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.981, $4.6764, 54938 output tokenscost per trial (USD)
flash 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0.957, $0.9611, 63274 output tokenst1_connectivity · usb_loose · trial 1: end score 1, $1.1015, 67834 output tokenst1_connectivity · usb_loose · trial 2: end score 0.952, $1.0895, 65943 output tokenst2_setup · missing_uneven · trial 0: end score 0.667, $0.5413, 37491 output tokenst2_setup · missing_uneven · trial 1: end score 0.929, $0.3247, 27233 output tokenst2_setup · missing_uneven · trial 2: end score 0.956, $0.3956, 40031 output tokenst3_light · two_targets_room_light · trial 0: end score 0, $1.3985, 68175 output tokenst3_light · two_targets_room_light · trial 1: end score 0.827, $1.0423, 53349 output tokenst3_light · two_targets_room_light · trial 2: end score 0.476, $1.5145, 71696 output tokenst4_takeover · t_present_faults · trial 0: end score 0, $1.0286, 69569 output tokenst4_takeover · t_present_faults · trial 1: end score 0, $0.7285, 59575 output tokenst4_takeover · t_present_faults · trial 2: end score 0, $1.0730, 67408 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.977, $0.7788, 67564 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.996, $0.6508, 57801 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.965, $1.1189, 79216 output tokenscost per trial (USD)
gemini_pro 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0, $1.3239, 11011 output tokenst1_connectivity · usb_loose · trial 1: end score 0.9, $2.7099, 44209 output tokenst1_connectivity · usb_loose · trial 2: end score 0.777, $1.3641, 22631 output tokenst2_setup · missing_uneven · trial 0: end score 0.8, $1.2646, 20542 output tokenst2_setup · missing_uneven · trial 1: end score 0.433, $0.2811, 8717 output tokenst2_setup · missing_uneven · trial 2: end score 0.899, $0.3164, 6246 output tokenst3_light · two_targets_room_light · trial 0: end score 0, $3.0036, 30929 output tokenst3_light · two_targets_room_light · trial 1: end score 0, $2.0642, 19815 output tokenst3_light · two_targets_room_light · trial 2: end score 0, $2.0381, 30813 output tokenst4_takeover · t_present_faults · trial 0: end score 0, $1.2168, 13638 output tokenst4_takeover · t_present_faults · trial 1: end score 0, $1.6610, 21892 output tokenst4_takeover · t_present_faults · trial 2: end score 0, $1.4100, 20767 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.713, $1.3438, 40229 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.903, $0.7540, 25034 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.979, $0.8793, 27826 output tokenscost per trial (USD)
haiku 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0.246, $0.0352, 25767 output tokenst1_connectivity · usb_loose · trial 1: end score 0.689, $0.8599, 120645 output tokenst1_connectivity · usb_loose · trial 2: end score 0.27, $0.0350, 28254 output tokenst2_setup · missing_uneven · trial 0: end score 0.981, $0.0739, 46915 output tokenst2_setup · missing_uneven · trial 1: end score 0.611, $0.0261, 16735 output tokenst2_setup · missing_uneven · trial 2: end score 0.91, $0.0770, 47051 output tokenst3_light · two_targets_room_light · trial 0: end score 0.83, $0.2470, 71335 output tokenst3_light · two_targets_room_light · trial 1: end score 0.667, $0.6243, 102492 output tokenst3_light · two_targets_room_light · trial 2: end score 0.807, $0.4329, 82371 output tokenst4_takeover · t_present_faults · trial 0: end score 0.694, $0.6829, 117960 output tokenst4_takeover · t_present_faults · trial 1: end score 0.854, $0.6637, 123364 output tokenst4_takeover · t_present_faults · trial 2: end score 0, $0.3852, 86032 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.951, $0.0917, 58335 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.802, $0.1937, 69727 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.981, $0.0451, 41475 output tokenscost per trial (USD)
luna 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0.737, $0.0590, 32128 output tokenst1_connectivity · usb_loose · trial 1: end score 0, $0.0358, 24881 output tokenst1_connectivity · usb_loose · trial 2: end score 0, $0.0375, 24903 output tokenst2_setup · missing_uneven · trial 0: end score 0.637, $0.0113, 7048 output tokenst2_setup · missing_uneven · trial 1: end score 0.445, $0.0193, 13550 output tokenst2_setup · missing_uneven · trial 2: end score 0.539, $0.0126, 9476 output tokenst3_light · two_targets_room_light · trial 0: end score 0, $0.0987, 38677 output tokenst3_light · two_targets_room_light · trial 1: end score 0, $0.0527, 25108 output tokenst3_light · two_targets_room_light · trial 2: end score 0, $0.0601, 39329 output tokenst4_takeover · t_present_faults · trial 0: end score 0, $0.0561, 36272 output tokenst4_takeover · t_present_faults · trial 1: end score 0, $0.0283, 26201 output tokenst4_takeover · t_present_faults · trial 2: end score 0, $0.0510, 36902 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.872, $0.0125, 13929 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.631, $0.0162, 16511 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.922, $0.0174, 16802 output tokenscost per trial (USD)
nemotron 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 0, $0.2059, 126173 output tokenst1_connectivity · usb_loose · trial 1: end score 0, $0.1445, 94089 output tokenst1_connectivity · usb_loose · trial 2: end score 0, $0.1383, 129258 output tokenst2_setup · missing_uneven · trial 0: end score 0.752, $0.0357, 29742 output tokenst2_setup · missing_uneven · trial 1: end score 0.759, $0.0327, 26461 output tokenst2_setup · missing_uneven · trial 2: end score 0.513, $0.0981, 87651 output tokenst3_light · two_targets_room_light · trial 0: end score 0, $0.2049, 100280 output tokenst3_light · two_targets_room_light · trial 1: end score 0, $0.2892, 272266 output tokenst3_light · two_targets_room_light · trial 2: end score 0.286, $0.2253, 110406 output tokenst4_takeover · t_present_faults · trial 0: end score 0, $0.5657, 42512 output tokenst4_takeover · t_present_faults · trial 1: end score 0, $0.2812, 194645 output tokenst4_takeover · t_present_faults · trial 2: end score 0, $0.2232, 80306 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.974, $0.1601, 128793 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.644, $0.3024, 180930 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.849, $0.2311, 154222 output tokenscost per trial (USD)
opus 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 1, $2.2085, 34269 output tokenst1_connectivity · usb_loose · trial 1: end score 0.879, $2.1553, 36115 output tokenst1_connectivity · usb_loose · trial 2: end score 0.957, $1.6860, 28914 output tokenst2_setup · missing_uneven · trial 0: end score 1, $0.5333, 8226 output tokenst2_setup · missing_uneven · trial 1: end score 1, $0.4478, 8441 output tokenst2_setup · missing_uneven · trial 2: end score 1, $0.5264, 8493 output tokenst3_light · two_targets_room_light · trial 0: end score 0.967, $4.0717, 55680 output tokenst3_light · two_targets_room_light · trial 1: end score 0.822, $4.1437, 60235 output tokenst3_light · two_targets_room_light · trial 2: end score 0.807, $3.8710, 59457 output tokenst4_takeover · t_present_faults · trial 0: end score 0.981, $2.3292, 33959 output tokenst4_takeover · t_present_faults · trial 1: end score 0.981, $1.9142, 33326 output tokenst4_takeover · t_present_faults · trial 2: end score 0.981, $1.6584, 24887 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.997, $0.6830, 16847 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.997, $0.5234, 12240 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.999, $0.7654, 19733 output tokenscost per trial (USD)
sol 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 1, $0.3707, 9246 output tokenst1_connectivity · usb_loose · trial 1: end score 1, $0.2684, 6454 output tokenst1_connectivity · usb_loose · trial 2: end score 1, $0.3347, 8199 output tokenst2_setup · missing_uneven · trial 0: end score 1, $0.2015, 5075 output tokenst2_setup · missing_uneven · trial 1: end score 0.973, $0.1427, 3982 output tokenst2_setup · missing_uneven · trial 2: end score 0.956, $0.1384, 3596 output tokenst3_light · two_targets_room_light · trial 0: end score 0.988, $0.5194, 12229 output tokenst3_light · two_targets_room_light · trial 1: end score 0.818, $0.6050, 13425 output tokenst3_light · two_targets_room_light · trial 2: end score 0.832, $0.5569, 12903 output tokenst4_takeover · t_present_faults · trial 0: end score 0.971, $0.3951, 10721 output tokenst4_takeover · t_present_faults · trial 1: end score 0.981, $0.3518, 8376 output tokenst4_takeover · t_present_faults · trial 2: end score 0, $0.2816, 6523 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.999, $0.3234, 13205 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.999, $0.2392, 10644 output tokenst5_traces · frozen_and_off_target · trial 2: end score 1, $0.3247, 12499 output tokenscost per trial (USD)
sonnet 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 1, $1.3509, 51328 output tokenst1_connectivity · usb_loose · trial 1: end score 1, $1.0612, 37883 output tokenst1_connectivity · usb_loose · trial 2: end score 1, $0.6740, 27696 output tokenst2_setup · missing_uneven · trial 0: end score 1, $0.6220, 20749 output tokenst2_setup · missing_uneven · trial 1: end score 0.926, $0.9698, 28757 output tokenst2_setup · missing_uneven · trial 2: end score 1, $0.2686, 10103 output tokenst3_light · two_targets_room_light · trial 0: end score 0.818, $3.3604, 100736 output tokenst3_light · two_targets_room_light · trial 1: end score 0.929, $2.4460, 76687 output tokenst3_light · two_targets_room_light · trial 2: end score 0.915, $2.0351, 63923 output tokenst4_takeover · t_present_faults · trial 0: end score 0.981, $1.5205, 55007 output tokenst4_takeover · t_present_faults · trial 1: end score 0.981, $1.3641, 49045 output tokenst4_takeover · t_present_faults · trial 2: end score 0.981, $1.1093, 39088 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.991, $0.4194, 22273 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.99, $0.3258, 17557 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.981, $0.4709, 23718 output tokenscost per trial (USD)
terra 15 scored trial(s)
00.510$5.656$11.313t1_connectivity · usb_loose · trial 0: end score 1, $0.1951, 12479 output tokenst1_connectivity · usb_loose · trial 1: end score 1, $0.1588, 10521 output tokenst1_connectivity · usb_loose · trial 2: end score 1, $0.1610, 8851 output tokenst2_setup · missing_uneven · trial 0: end score 0.973, $0.0926, 4461 output tokenst2_setup · missing_uneven · trial 1: end score 0.981, $0.1844, 11092 output tokenst2_setup · missing_uneven · trial 2: end score 0.956, $0.1608, 7921 output tokenst3_light · two_targets_room_light · trial 0: end score 0.98, $0.2793, 14701 output tokenst3_light · two_targets_room_light · trial 1: end score 0.948, $0.2580, 12904 output tokenst3_light · two_targets_room_light · trial 2: end score 0, $0.2841, 12450 output tokenst4_takeover · t_present_faults · trial 0: end score 1, $0.2424, 12749 output tokenst4_takeover · t_present_faults · trial 1: end score 1, $0.2440, 13625 output tokenst4_takeover · t_present_faults · trial 2: end score 0.981, $0.2122, 12890 output tokenst5_traces · frozen_and_off_target · trial 0: end score 0.998, $0.1809, 14786 output tokenst5_traces · frozen_and_off_target · trial 1: end score 0.998, $0.1275, 10861 output tokenst5_traces · frozen_and_off_target · trial 2: end score 0.997, $0.1760, 13861 output tokenscost per trial (USD)