This page: the 250 scored episodes (195 headline + 55 ablations), total cost $314.15 (headline $247.81 + ablations $66.34) · 465 more in the Sections above: E1 stress 48 · E2 interlock 32 · E3 LED wording 18 · set-aside attempts 19 · development history 218 · E5 sweep 130 · every record is also in the dataset's episodes table · click a trial to step through it
| agent | task | trials | harness errors | submitted | all pass | mean end score | cost | cost / trial |
|---|---|---|---|---|---|---|---|---|
| argon | t1_connectivity | 3 | — | 3 | 1 / 3 | 0.956 | $5.5327 | $1.8442 |
| argon | t2_setup | 3 | — | 3 | 2 / 3 | 0.991 | $1.0146 | $0.3382 |
| argon | t3_light | 3 | — | 3 | 1 / 3 | 0.831 | $8.6109 | $2.8703 |
| argon | t4_takeover | 3 | — | 3 | 0 / 3 | 0.946 | $6.5164 | $2.1721 |
| argon | t5_traces | 3 | — | 3 | 3 / 3 | 0.999 | $7.4622 | $2.4874 |
| argon | all tasks | 15 | — | 15 | 7 / 15 | 0.945 | $29.1367 | $1.9424 |
| astra | t1_connectivity | 3 | — | 3 | 2 / 3 | 0.986 | $6.7555 | $2.2518 |
| astra | t2_setup | 3 | — | 3 | 0 / 3 | 0.96 | $2.7812 | $0.9271 |
| astra | t3_light | 3 | — | 3 | 1 / 3 | 0.319 | $11.8218 | $3.9406 |
| astra | t4_takeover | 3 | — | 3 | 2 / 3 | 0.667 | $8.2688 | $2.7563 |
| astra | t5_traces | 3 | — | 3 | 3 / 3 | 0.999 | $5.3782 | $1.7927 |
| astra | all tasks | 15 | — | 15 | 8 / 15 | 0.786 | $35.0056 | $2.3337 |
| deepseek | t1_connectivity | 3 | — | 3 | 0 / 3 | 0.144 | $0.2752 | $0.0917 |
| deepseek | t2_setup | 3 | — | 3 | 0 / 3 | 0.941 | $0.1299 | $0.0433 |
| deepseek | t3_light | 3 | — | 3 | 0 / 3 | 0 | $0.2845 | $0.0948 |
| deepseek | t4_takeover | 3 | — | 2 | 0 / 3 | 0.319 | $0.3161 | $0.1054 |
| deepseek | t5_traces | 3 | — | 3 | 3 / 3 | 0.996 | $0.2189 | $0.0730 |
| deepseek | all tasks | 15 | — | 14 | 3 / 15 | 0.48 | $1.2246 | $0.0816 |
| fable | t1_connectivity | 4 | — | 4 | 2 / 4 | 0.936 | $19.1200 | $4.7800 |
| fable | t2_setup | 4 | — | 4 | 4 / 4 | 1 | $9.4116 | $2.3529 |
| fable | t3_light | 4 | — | 4 | 3 / 4 | 0.922 | $33.5165 | $8.3791 |
| fable | t4_takeover | 4 | — | 4 | 0 / 4 | 0.965 | $31.2461 | $7.8115 |
| fable | t5_traces | 4 | — | 4 | 3 / 4 | 0.991 | $23.6599 | $5.9150 |
| fable | all tasks | 20 | — | 20 | 12 / 20 | 0.963 | $116.9541 | $5.8477 |
| flash | t1_connectivity | 5 | — | 5 | 2 / 5 | 0.94 | $5.3697 | $1.0739 |
| flash | t2_setup | 5 | — | 5 | 0 / 5 | 0.893 | $2.0172 | $0.4034 |
| flash | t3_light | 5 | — | 4 | 0 / 5 | 0.394 | $7.4015 | $1.4803 |
| flash | t4_takeover | 5 | — | 5 | 0 / 5 | 0.178 | $4.4751 | $0.8950 |
| flash | t5_traces | 5 | — | 5 | 2 / 5 | 0.975 | $4.2017 | $0.8403 |
| flash | all tasks | 25 | — | 24 | 4 / 25 | 0.676 | $23.4653 | $0.9386 |
| gemini_pro | t1_connectivity | 3 | — | 3 | 0 / 3 | 0.559 | $5.3979 | $1.7993 |
| gemini_pro | t2_setup | 3 | — | 3 | 0 / 3 | 0.711 | $1.8620 | $0.6207 |
| gemini_pro | t3_light | 3 | — | 2 | 0 / 3 | 0 | $7.1060 | $2.3687 |
| gemini_pro | t4_takeover | 3 | — | 3 | 0 / 3 | 0 | $4.2879 | $1.4293 |
| gemini_pro | t5_traces | 3 | — | 3 | 1 / 3 | 0.865 | $2.9771 | $0.9924 |
| gemini_pro | all tasks | 15 | — | 14 | 1 / 15 | 0.427 | $21.6308 | $1.4421 |
| haiku | t1_connectivity | 5 | — | 5 | 0 / 5 | 0.382 | $1.6794 | $0.3359 |
| haiku | t2_setup | 5 | — | 5 | 1 / 5 | 0.891 | $0.2689 | $0.0538 |
| haiku | t3_light | 5 | — | 5 | 0 / 5 | 0.461 | $2.4063 | $0.4813 |
| haiku | t4_takeover | 5 | — | 5 | 0 / 5 | 0.651 | $2.7888 | $0.5578 |
| haiku | t5_traces | 5 | — | 5 | 1 / 5 | 0.932 | $0.4488 | $0.0898 |
| haiku | all tasks | 25 | — | 25 | 2 / 25 | 0.663 | $7.5923 | $0.3037 |
| luna | t1_connectivity | 3 | — | 3 | 0 / 3 | 0.246 | $0.1323 | $0.0441 |
| luna | t2_setup | 3 | — | 3 | 0 / 3 | 0.541 | $0.0432 | $0.0144 |
| luna | t3_light | 3 | — | 3 | 0 / 3 | 0 | $0.2115 | $0.0705 |
| luna | t4_takeover | 3 | — | 3 | 0 / 3 | 0 | $0.1355 | $0.0452 |
| luna | t5_traces | 3 | — | 3 | 0 / 3 | 0.808 | $0.0461 | $0.0154 |
| luna | all tasks | 15 | — | 15 | 0 / 15 | 0.319 | $0.5686 | $0.0379 |
| nemotron | t1_connectivity | 3 | — | 3 | 0 / 3 | 0 | $0.4886 | $0.1629 |
| nemotron | t2_setup | 3 | — | 3 | 0 / 3 | 0.675 | $0.1665 | $0.0555 |
| nemotron | t3_light | 3 | — | 3 | 0 / 3 | 0.095 | $0.7194 | $0.2398 |
| nemotron | t4_takeover | 3 | — | 2 | 0 / 3 | 0 | $1.0701 | $0.3567 |
| nemotron | t5_traces | 3 | — | 3 | 0 / 3 | 0.822 | $0.6936 | $0.2312 |
| nemotron | all tasks | 15 | — | 14 | 0 / 15 | 0.318 | $3.1382 | $0.2092 |
| opus | t1_connectivity | 5 | — | 5 | 3 / 5 | 0.967 | $9.0085 | $1.8017 |
| opus | t2_setup | 5 | — | 5 | 5 / 5 | 1 | $2.5008 | $0.5002 |
| opus | t3_light | 5 | — | 5 | 1 / 5 | 0.848 | $18.0180 | $3.6036 |
| opus | t4_takeover | 5 | — | 5 | 0 / 5 | 0.981 | $10.2705 | $2.0541 |
| opus | t5_traces | 5 | — | 5 | 5 / 5 | 0.998 | $4.7860 | $0.9572 |
| opus | all tasks | 25 | — | 25 | 14 / 25 | 0.959 | $44.5838 | $1.7834 |
| sol | t1_connectivity | 5 | — | 5 | 5 / 5 | 1 | $1.5804 | $0.3161 |
| sol | t2_setup | 5 | — | 5 | 3 / 5 | 0.986 | $0.7223 | $0.1445 |
| sol | t3_light | 5 | — | 5 | 1 / 5 | 0.85 | $2.5633 | $0.5127 |
| sol | t4_takeover | 5 | — | 5 | 2 / 5 | 0.79 | $1.8087 | $0.3617 |
| sol | t5_traces | 5 | — | 5 | 5 / 5 | 0.998 | $1.5209 | $0.3042 |
| sol | all tasks | 25 | — | 25 | 16 / 25 | 0.925 | $8.1955 | $0.3278 |
| sonnet | t1_connectivity | 3 | — | 3 | 3 / 3 | 1 | $3.0861 | $1.0287 |
| sonnet | t2_setup | 3 | — | 3 | 2 / 3 | 0.975 | $1.8605 | $0.6202 |
| sonnet | t3_light | 3 | — | 3 | 0 / 3 | 0.887 | $7.8415 | $2.6138 |
| sonnet | t4_takeover | 3 | — | 3 | 0 / 3 | 0.981 | $3.9939 | $1.3313 |
| sonnet | t5_traces | 3 | — | 3 | 2 / 3 | 0.987 | $1.2161 | $0.4054 |
| sonnet | all tasks | 15 | — | 15 | 7 / 15 | 0.966 | $17.9981 | $1.1999 |
| terra | t1_connectivity | 5 | — | 5 | 4 / 5 | 0.991 | $0.8671 | $0.1734 |
| terra | t2_setup | 5 | — | 5 | 0 / 5 | 0.971 | $0.6387 | $0.1277 |
| terra | t3_light | 5 | — | 5 | 1 / 5 | 0.739 | $1.2369 | $0.2474 |
| terra | t4_takeover | 5 | — | 5 | 4 / 5 | 0.996 | $1.1074 | $0.2215 |
| terra | t5_traces | 5 | — | 5 | 5 / 5 | 0.998 | $0.8092 | $0.1618 |
| terra | all tasks | 25 | — | 25 | 14 / 25 | 0.939 | $4.6593 | $0.1864 |
Each trial has one end grade, computed once after the run ends. Harness failures (the run stopped on a provider or harness error) are listed but have no score: they are left out of "all pass" and the means.
| headline | argon | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $1.7631 | 25 | 43 |
| headline | argon | t1_connectivity | usb_loose | trial 1 | submitted | 0.9167 | fail | $2.0416 | 34 | 93 |
| headline | argon | t1_connectivity | usb_loose | trial 2 | submitted | 0.952 | fail | $1.7280 | 23 | 53 |
| headline | astra | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $2.2998 | 32 | 50 |
| headline | astra | t1_connectivity | usb_loose | trial 1 | submitted | 1 | pass | $2.3290 | 36 | 55 |
| headline | astra | t1_connectivity | usb_loose | trial 2 | submitted | 0.957 | fail | $2.1267 | 35 | 56 |
| headline | deepseek | t1_connectivity | usb_loose | trial 0 | submitted | 0.432 | fail | $0.0767 | 44 | 101 |
| headline | deepseek | t1_connectivity | usb_loose | trial 1 | submitted | 0 | fail | $0.1010 | 48 | 92 |
| headline | deepseek | t1_connectivity | usb_loose | trial 2 | submitted | 0 | fail | $0.0974 | 52 | 103 |
| headline | fable | t1_connectivity | usb_loose | trial 0 | submitted | 0.791 | fail | $5.9012 | 23 | 61 |
| headline | fable | t1_connectivity | usb_loose | trial 1 | submitted | 0.952 | fail | $2.8121 | 16 | 43 |
| headline | fable | t1_connectivity | usb_loose | trial 2 | submitted | 1 | pass | $5.7601 | 23 | 65 |
| minimal | fable | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $4.6466 | 22 | 60 |
| contract_only | flash | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $1.2312 | 30 | 108 |
| headline | flash | t1_connectivity | usb_loose | trial 0 | submitted | 0.957 | fail | $0.9611 | 24 | 78 |
| headline | flash | t1_connectivity | usb_loose | trial 1 | submitted | 1 | pass | $1.1015 | 30 | 94 |
| headline | flash | t1_connectivity | usb_loose | trial 2 | submitted | 0.952 | fail | $1.0895 | 47 | 104 |
| minimal | flash | t1_connectivity | usb_loose | trial 0 | submitted | 0.791 | fail | $0.9864 | 39 | 105 |
| headline | gemini_pro | t1_connectivity | usb_loose | trial 0 | submitted | 0 | fail | $1.3239 | 26 | 71 |
| headline | gemini_pro | t1_connectivity | usb_loose | trial 1 | submitted | 0.9 | fail | $2.7099 | 30 | 101 |
| headline | gemini_pro | t1_connectivity | usb_loose | trial 2 | submitted | 0.777 | fail | $1.3641 | 23 | 62 |
| contract_only | haiku | t1_connectivity | usb_loose | trial 0 | submitted | 0 | fail | $0.0396 | 16 | 42 |
| headline | haiku | t1_connectivity | usb_loose | trial 0 | submitted | 0.246 | fail | $0.0352 | 19 | 38 |
| headline | haiku | t1_connectivity | usb_loose | trial 1 | submitted | 0.689 | fail | $0.8599 | 51 | 125 |
| headline | haiku | t1_connectivity | usb_loose | trial 2 | submitted | 0.27 | fail | $0.0350 | 18 | 38 |
| minimal | haiku | t1_connectivity | usb_loose | trial 0 | submitted | 0.707 | fail | $0.7097 | 48 | 125 |
| headline | luna | t1_connectivity | usb_loose | trial 0 | submitted | 0.737 | fail | $0.0590 | 30 | 56 |
| headline | luna | t1_connectivity | usb_loose | trial 1 | submitted | 0 | fail | $0.0358 | 30 | 55 |
| headline | luna | t1_connectivity | usb_loose | trial 2 | submitted | 0 | fail | $0.0375 | 27 | 49 |
| headline | nemotron | t1_connectivity | usb_loose | trial 0 | submitted | 0 | fail | $0.2059 | 29 | 66 |
| headline | nemotron | t1_connectivity | usb_loose | trial 1 | submitted | 0 | fail | $0.1445 | 24 | 63 |
| headline | nemotron | t1_connectivity | usb_loose | trial 2 | submitted | 0 | fail | $0.1383 | 23 | 43 |
| contract_only | opus | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $1.5794 | 21 | 54 |
| headline | opus | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $2.2085 | 31 | 70 |
| headline | opus | t1_connectivity | usb_loose | trial 1 | submitted | 0.879 | fail | $2.1553 | 40 | 75 |
| headline | opus | t1_connectivity | usb_loose | trial 2 | submitted | 0.957 | fail | $1.6860 | 28 | 62 |
| minimal | opus | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $1.3793 | 35 | 59 |
| contract_only | sol | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.2332 | 27 | 43 |
| headline | sol | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.3707 | 44 | 69 |
| headline | sol | t1_connectivity | usb_loose | trial 1 | submitted | 1 | pass | $0.2684 | 33 | 50 |
| headline | sol | t1_connectivity | usb_loose | trial 2 | submitted | 1 | pass | $0.3347 | 34 | 55 |
| minimal | sol | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.3735 | 38 | 66 |
| headline | sonnet | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $1.3509 | 40 | 91 |
| headline | sonnet | t1_connectivity | usb_loose | trial 1 | submitted | 1 | pass | $1.0612 | 41 | 76 |
| headline | sonnet | t1_connectivity | usb_loose | trial 2 | submitted | 1 | pass | $0.6740 | 25 | 51 |
| contract_only | terra | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.1360 | 22 | 41 |
| headline | terra | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.1951 | 33 | 57 |
| headline | terra | t1_connectivity | usb_loose | trial 1 | submitted | 1 | pass | $0.1588 | 23 | 45 |
| headline | terra | t1_connectivity | usb_loose | trial 2 | submitted | 1 | pass | $0.1610 | 30 | 48 |
| minimal | terra | t1_connectivity | usb_loose | trial 0 | submitted | 0.957 | fail | $0.2160 | 30 | 55 |
| headline | argon | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.3430 | 16 | 20 |
| headline | argon | t2_setup | missing_uneven | trial 1 | submitted | 0.973 | fail | $0.3797 | 16 | 26 |
| headline | argon | t2_setup | missing_uneven | trial 2 | submitted | 1 | pass | $0.2919 | 16 | 24 |
| headline | astra | t2_setup | missing_uneven | trial 0 | submitted | 0.973 | fail | $0.9051 | 16 | 26 |
| headline | astra | t2_setup | missing_uneven | trial 1 | submitted | 0.933 | fail | $1.0786 | 17 | 29 |
| headline | astra | t2_setup | missing_uneven | trial 2 | submitted | 0.973 | fail | $0.7975 | 15 | 24 |
| headline | deepseek | t2_setup | missing_uneven | trial 0 | submitted | 0.933 | fail | $0.0353 | 18 | 48 |
| headline | deepseek | t2_setup | missing_uneven | trial 1 | submitted | 0.917 | fail | $0.0439 | 18 | 46 |
| headline | deepseek | t2_setup | missing_uneven | trial 2 | submitted | 0.973 | fail | $0.0508 | 16 | 49 |
| headline | fable | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $2.5024 | 17 | 36 |
| headline | fable | t2_setup | missing_uneven | trial 1 | submitted | 1 | pass | $2.2119 | 14 | 28 |
| headline | fable | t2_setup | missing_uneven | trial 2 | submitted | 1 | pass | $2.2961 | 15 | 30 |
| minimal | fable | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $2.4012 | 15 | 33 |
| contract_only | flash | t2_setup | missing_uneven | trial 0 | submitted | 0.956 | fail | $0.3538 | 19 | 48 |
| headline | flash | t2_setup | missing_uneven | trial 0 | submitted | 0.667 | fail | $0.5413 | 23 | 57 |
| headline | flash | t2_setup | missing_uneven | trial 1 | submitted | 0.929 | fail | $0.3247 | 17 | 42 |
| headline | flash | t2_setup | missing_uneven | trial 2 | submitted | 0.956 | fail | $0.3956 | 17 | 45 |
| minimal | flash | t2_setup | missing_uneven | trial 0 | submitted | 0.956 | fail | $0.4017 | 16 | 52 |
| headline | gemini_pro | t2_setup | missing_uneven | trial 0 | submitted | 0.8 | fail | $1.2646 | 53 | 74 |
| headline | gemini_pro | t2_setup | missing_uneven | trial 1 | submitted | 0.433 | fail | $0.2811 | 21 | 25 |
| headline | gemini_pro | t2_setup | missing_uneven | trial 2 | submitted | 0.899 | fail | $0.3164 | 22 | 27 |
| contract_only | haiku | t2_setup | missing_uneven | trial 0 | submitted | 0.956 | fail | $0.0232 | 19 | 34 |
| headline | haiku | t2_setup | missing_uneven | trial 0 | submitted | 0.981 | fail | $0.0739 | 31 | 64 |
| headline | haiku | t2_setup | missing_uneven | trial 1 | submitted | 0.611 | fail | $0.0261 | 23 | 38 |
| headline | haiku | t2_setup | missing_uneven | trial 2 | submitted | 0.91 | fail | $0.0770 | 34 | 63 |
| minimal | haiku | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.0688 | 28 | 59 |
| headline | luna | t2_setup | missing_uneven | trial 0 | submitted | 0.637 | fail | $0.0113 | 16 | 27 |
| headline | luna | t2_setup | missing_uneven | trial 1 | submitted | 0.445 | fail | $0.0193 | 25 | 41 |
| headline | luna | t2_setup | missing_uneven | trial 2 | submitted | 0.539 | fail | $0.0126 | 19 | 28 |
| headline | nemotron | t2_setup | missing_uneven | trial 0 | submitted | 0.752 | fail | $0.0357 | 16 | 21 |
| headline | nemotron | t2_setup | missing_uneven | trial 1 | submitted | 0.759 | fail | $0.0327 | 15 | 25 |
| headline | nemotron | t2_setup | missing_uneven | trial 2 | submitted | 0.513 | fail | $0.0981 | 24 | 47 |
| contract_only | opus | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.5289 | 12 | 25 |
| headline | opus | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.5333 | 13 | 26 |
| headline | opus | t2_setup | missing_uneven | trial 1 | submitted | 1 | pass | $0.4478 | 12 | 24 |
| headline | opus | t2_setup | missing_uneven | trial 2 | submitted | 1 | pass | $0.5264 | 14 | 26 |
| minimal | opus | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.4643 | 13 | 24 |
| contract_only | sol | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.1076 | 15 | 24 |
| headline | sol | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.2015 | 32 | 42 |
| headline | sol | t2_setup | missing_uneven | trial 1 | submitted | 0.973 | fail | $0.1427 | 16 | 27 |
| headline | sol | t2_setup | missing_uneven | trial 2 | submitted | 0.956 | fail | $0.1384 | 17 | 26 |
| minimal | sol | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.1322 | 14 | 25 |
| headline | sonnet | t2_setup | missing_uneven | trial 0 | submitted | 1 | pass | $0.6220 | 22 | 42 |
| headline | sonnet | t2_setup | missing_uneven | trial 1 | submitted | 0.926 | fail | $0.9698 | 30 | 56 |
| headline | sonnet | t2_setup | missing_uneven | trial 2 | submitted | 1 | pass | $0.2686 | 15 | 28 |
| contract_only | terra | t2_setup | missing_uneven | trial 0 | submitted | 0.973 | fail | $0.1046 | 19 | 33 |
| headline | terra | t2_setup | missing_uneven | trial 0 | submitted | 0.973 | fail | $0.0926 | 18 | 28 |
| headline | terra | t2_setup | missing_uneven | trial 1 | submitted | 0.981 | fail | $0.1844 | 24 | 39 |
| headline | terra | t2_setup | missing_uneven | trial 2 | submitted | 0.956 | fail | $0.1608 | 18 | 32 |
| minimal | terra | t2_setup | missing_uneven | trial 0 | submitted | 0.973 | fail | $0.0962 | 18 | 29 |
| headline | argon | t3_light | two_targets_room_light | trial 0 | submitted | 1 | pass | $2.6245 | 42 | 116 |
| headline | argon | t3_light | two_targets_room_light | trial 1 | submitted | 0.828 | fail | $3.4773 | 52 | 68 |
| headline | argon | t3_light | two_targets_room_light | trial 2 | submitted | 0.667 | fail | $2.5090 | 50 | 126 |
| headline | astra | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $5.2746 | 61 | 107 |
| headline | astra | t3_light | two_targets_room_light | trial 1 | submitted | 0 | fail | $1.5741 | 33 | 48 |
| headline | astra | t3_light | two_targets_room_light | trial 2 | submitted | 0.956 | pass | $4.9731 | 69 | 101 |
| headline | deepseek | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $0.1154 | 68 | 134 |
| headline | deepseek | t3_light | two_targets_room_light | trial 1 | submitted | 0 | fail | $0.0689 | 37 | 68 |
| headline | deepseek | t3_light | two_targets_room_light | trial 2 | submitted | 0 | fail | $0.1003 | 99 | 148 |
| headline | fable | t3_light | two_targets_room_light | trial 0 | submitted | 0.802 | fail | $10.7740 | 59 | 130 |
| headline | fable | t3_light | two_targets_room_light | trial 1 | submitted | 0.962 | pass | $7.5294 | 40 | 93 |
| headline | fable | t3_light | two_targets_room_light | trial 2 | submitted | 0.972 | pass | $6.4541 | 42 | 80 |
| minimal | fable | t3_light | two_targets_room_light | trial 0 | submitted | 0.954 | pass | $8.7590 | 48 | 115 |
| contract_only | flash | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $1.7352 | 91 | 159 |
| headline | flash | t3_light | two_targets_room_light | trial 0 | no submission | 0 | fail | $1.3985 | 100 | 144 |
| headline | flash | t3_light | two_targets_room_light | trial 1 | submitted | 0.827 | fail | $1.0423 | 66 | 114 |
| headline | flash | t3_light | two_targets_room_light | trial 2 | submitted | 0.476 | fail | $1.5145 | 75 | 129 |
| minimal | flash | t3_light | two_targets_room_light | trial 0 | submitted | 0.667 | fail | $1.7111 | 83 | 143 |
| headline | gemini_pro | t3_light | two_targets_room_light | trial 0 | no submission | 0 | fail | $3.0036 | 100 | 176 |
| headline | gemini_pro | t3_light | two_targets_room_light | trial 1 | submitted | 0 | fail | $2.0642 | 98 | 143 |
| headline | gemini_pro | t3_light | two_targets_room_light | trial 2 | submitted | 0 | fail | $2.0381 | 54 | 113 |
| contract_only | haiku | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $0.3184 | 51 | 109 |
| headline | haiku | t3_light | two_targets_room_light | trial 0 | submitted | 0.83 | fail | $0.2470 | 47 | 98 |
| headline | haiku | t3_light | two_targets_room_light | trial 1 | submitted | 0.667 | fail | $0.6243 | 66 | 120 |
| headline | haiku | t3_light | two_targets_room_light | trial 2 | submitted | 0.807 | fail | $0.4329 | 58 | 118 |
| minimal | haiku | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $0.7837 | 81 | 157 |
| headline | luna | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $0.0987 | 92 | 144 |
| headline | luna | t3_light | two_targets_room_light | trial 1 | submitted | 0 | fail | $0.0527 | 56 | 83 |
| headline | luna | t3_light | two_targets_room_light | trial 2 | submitted | 0 | fail | $0.0601 | 65 | 97 |
| headline | nemotron | t3_light | two_targets_room_light | trial 0 | submitted | 0 | fail | $0.2049 | 75 | 112 |
| headline | nemotron | t3_light | two_targets_room_light | trial 1 | submitted | 0 | fail | $0.2892 | 49 | 85 |
| headline | nemotron | t3_light | two_targets_room_light | trial 2 | submitted | 0.286 | fail | $0.2253 | 48 | 88 |
| contract_only | opus | t3_light | two_targets_room_light | trial 0 | submitted | 0.823 | fail | $2.6207 | 52 | 104 |
| headline | opus | t3_light | two_targets_room_light | trial 0 | submitted | 0.967 | pass | $4.0717 | 64 | 139 |
| headline | opus | t3_light | two_targets_room_light | trial 1 | submitted | 0.822 | fail | $4.1437 | 78 | 150 |
| headline | opus | t3_light | two_targets_room_light | trial 2 | submitted | 0.807 | fail | $3.8710 | 60 | 122 |
| minimal | opus | t3_light | two_targets_room_light | trial 0 | submitted | 0.823 | fail | $3.3108 | 59 | 119 |
| contract_only | sol | t3_light | two_targets_room_light | trial 0 | submitted | 0.783 | fail | $0.5412 | 51 | 87 |
| headline | sol | t3_light | two_targets_room_light | trial 0 | submitted | 0.988 | pass | $0.5194 | 54 | 89 |
| headline | sol | t3_light | two_targets_room_light | trial 1 | submitted | 0.818 | fail | $0.6050 | 49 | 84 |
| headline | sol | t3_light | two_targets_room_light | trial 2 | submitted | 0.832 | fail | $0.5569 | 47 | 89 |
| minimal | sol | t3_light | two_targets_room_light | trial 0 | submitted | 0.83 | fail | $0.3408 | 36 | 72 |
| headline | sonnet | t3_light | two_targets_room_light | trial 0 | submitted | 0.818 | fail | $3.3604 | 79 | 181 |
| headline | sonnet | t3_light | two_targets_room_light | trial 1 | submitted | 0.929 | fail | $2.4460 | 82 | 153 |
| headline | sonnet | t3_light | two_targets_room_light | trial 2 | submitted | 0.915 | fail | $2.0351 | 50 | 117 |
| contract_only | terra | t3_light | two_targets_room_light | trial 0 | submitted | 0.932 | fail | $0.1747 | 31 | 56 |
| headline | terra | t3_light | two_targets_room_light | trial 0 | submitted | 0.98 | pass | $0.2793 | 39 | 74 |
| headline | terra | t3_light | two_targets_room_light | trial 1 | submitted | 0.948 | fail | $0.2580 | 36 | 75 |
| headline | terra | t3_light | two_targets_room_light | trial 2 | submitted | 0 | fail | $0.2841 | 39 | 70 |
| minimal | terra | t3_light | two_targets_room_light | trial 0 | submitted | 0.833 | fail | $0.2407 | 37 | 69 |
| headline | argon | t4_takeover | t_present_faults | trial 0 | submitted | 0.927 | fail | $1.7729 | 19 | 81 |
| headline | argon | t4_takeover | t_present_faults | trial 1 | submitted | 0.956 | fail | $2.4672 | 19 | 64 |
| headline | argon | t4_takeover | t_present_faults | trial 2 | submitted | 0.955 | fail | $2.2763 | 20 | 53 |
| headline | astra | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $3.0083 | 35 | 59 |
| headline | astra | t4_takeover | t_present_faults | trial 1 | submitted | 0 | fail | $1.6005 | 17 | 31 |
| headline | astra | t4_takeover | t_present_faults | trial 2 | submitted | 1 | pass | $3.6599 | 36 | 63 |
| headline | deepseek | t4_takeover | t_present_faults | trial 0 | no submission | 0 | fail | $0.1307 | 60 | 149 |
| headline | deepseek | t4_takeover | t_present_faults | trial 1 | submitted | 0 | fail | $0.0902 | 21 | 95 |
| headline | deepseek | t4_takeover | t_present_faults | trial 2 | submitted | 0.956 | fail | $0.0951 | 22 | 89 |
| headline | fable | t4_takeover | t_present_faults | trial 0 | submitted | 0.953 | fail | $9.4815 | 24 | 89 |
| headline | fable | t4_takeover | t_present_faults | trial 1 | submitted | 0.981 | fail | $7.1882 | 21 | 78 |
| headline | fable | t4_takeover | t_present_faults | trial 2 | submitted | 0.974 | fail | $7.5607 | 22 | 74 |
| minimal | fable | t4_takeover | t_present_faults | trial 0 | submitted | 0.953 | fail | $7.0157 | 27 | 76 |
| contract_only | flash | t4_takeover | t_present_faults | trial 0 | submitted | 0.892 | fail | $0.8898 | 25 | 77 |
| headline | flash | t4_takeover | t_present_faults | trial 0 | submitted | 0 | fail | $1.0286 | 28 | 88 |
| headline | flash | t4_takeover | t_present_faults | trial 1 | submitted | 0 | fail | $0.7285 | 15 | 65 |
| headline | flash | t4_takeover | t_present_faults | trial 2 | submitted | 0 | fail | $1.0730 | 28 | 103 |
| minimal | flash | t4_takeover | t_present_faults | trial 0 | submitted | 0 | fail | $0.7551 | 26 | 72 |
| headline | gemini_pro | t4_takeover | t_present_faults | trial 0 | submitted | 0 | fail | $1.2168 | 37 | 80 |
| headline | gemini_pro | t4_takeover | t_present_faults | trial 1 | submitted | 0 | fail | $1.6610 | 44 | 106 |
| headline | gemini_pro | t4_takeover | t_present_faults | trial 2 | submitted | 0 | fail | $1.4100 | 43 | 83 |
| contract_only | haiku | t4_takeover | t_present_faults | trial 0 | submitted | 0.924 | fail | $0.5961 | 43 | 104 |
| headline | haiku | t4_takeover | t_present_faults | trial 0 | submitted | 0.694 | fail | $0.6829 | 42 | 107 |
| headline | haiku | t4_takeover | t_present_faults | trial 1 | submitted | 0.854 | fail | $0.6637 | 42 | 105 |
| headline | haiku | t4_takeover | t_present_faults | trial 2 | submitted | 0 | fail | $0.3852 | 38 | 84 |
| minimal | haiku | t4_takeover | t_present_faults | trial 0 | submitted | 0.782 | fail | $0.4608 | 46 | 110 |
| headline | luna | t4_takeover | t_present_faults | trial 0 | submitted | 0 | fail | $0.0561 | 31 | 65 |
| headline | luna | t4_takeover | t_present_faults | trial 1 | submitted | 0 | fail | $0.0283 | 13 | 37 |
| headline | luna | t4_takeover | t_present_faults | trial 2 | submitted | 0 | fail | $0.0510 | 25 | 65 |
| headline | nemotron | t4_takeover | t_present_faults | trial 0 | no submission | 0 | fail | $0.5657 | 1 | 180 |
| headline | nemotron | t4_takeover | t_present_faults | trial 1 | submitted | 0 | fail | $0.2812 | 36 | 77 |
| headline | nemotron | t4_takeover | t_present_faults | trial 2 | submitted | 0 | fail | $0.2232 | 19 | 60 |
| contract_only | opus | t4_takeover | t_present_faults | trial 0 | submitted | 0.981 | fail | $2.3129 | 31 | 75 |
| headline | opus | t4_takeover | t_present_faults | trial 0 | submitted | 0.981 | fail | $2.3292 | 31 | 70 |
| headline | opus | t4_takeover | t_present_faults | trial 1 | submitted | 0.981 | fail | $1.9142 | 18 | 61 |
| headline | opus | t4_takeover | t_present_faults | trial 2 | submitted | 0.981 | fail | $1.6584 | 22 | 54 |
| minimal | opus | t4_takeover | t_present_faults | trial 0 | submitted | 0.981 | fail | $2.0558 | 24 | 67 |
| contract_only | sol | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $0.3960 | 30 | 55 |
| headline | sol | t4_takeover | t_present_faults | trial 0 | submitted | 0.971 | fail | $0.3951 | 31 | 58 |
| headline | sol | t4_takeover | t_present_faults | trial 1 | submitted | 0.981 | fail | $0.3518 | 34 | 55 |
| headline | sol | t4_takeover | t_present_faults | trial 2 | submitted | 0 | fail | $0.2816 | 26 | 46 |
| minimal | sol | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $0.3841 | 32 | 60 |
| headline | sonnet | t4_takeover | t_present_faults | trial 0 | submitted | 0.981 | fail | $1.5205 | 33 | 83 |
| headline | sonnet | t4_takeover | t_present_faults | trial 1 | submitted | 0.981 | fail | $1.3641 | 31 | 86 |
| headline | sonnet | t4_takeover | t_present_faults | trial 2 | submitted | 0.981 | fail | $1.1093 | 20 | 68 |
| contract_only | terra | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $0.1876 | 22 | 46 |
| headline | terra | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $0.2424 | 19 | 42 |
| headline | terra | t4_takeover | t_present_faults | trial 1 | submitted | 1 | pass | $0.2440 | 24 | 52 |
| headline | terra | t4_takeover | t_present_faults | trial 2 | submitted | 0.981 | fail | $0.2122 | 29 | 53 |
| minimal | terra | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $0.2213 | 23 | 45 |
| headline | argon | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.996 | pass | $1.5369 | 1 | 62 |
| headline | argon | t5_traces | frozen_and_off_target | trial 1 | submitted | 1 | pass | $3.6185 | 1 | 72 |
| headline | argon | t5_traces | frozen_and_off_target | trial 2 | submitted | 1 | pass | $2.3068 | 1 | 60 |
| headline | astra | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $1.4137 | 1 | 20 |
| headline | astra | t5_traces | frozen_and_off_target | trial 1 | submitted | 1 | pass | $2.5409 | 1 | 33 |
| headline | astra | t5_traces | frozen_and_off_target | trial 2 | submitted | 1 | pass | $1.4236 | 1 | 18 |
| headline | deepseek | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.995 | pass | $0.0782 | 1 | 83 |
| headline | deepseek | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.996 | pass | $0.0682 | 1 | 78 |
| headline | deepseek | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.997 | pass | $0.0725 | 1 | 74 |
| headline | fable | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $4.8039 | 1 | 32 |
| headline | fable | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.995 | pass | $5.4050 | 1 | 42 |
| headline | fable | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.981 | pass | $4.6764 | 1 | 34 |
| minimal | fable | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.991 | fail | $8.7747 | 1 | 59 |
| contract_only | flash | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.967 | fail | $0.6856 | 1 | 61 |
| headline | flash | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.977 | pass | $0.7788 | 1 | 71 |
| headline | flash | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.996 | pass | $0.6508 | 1 | 74 |
| headline | flash | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.965 | fail | $1.1189 | 1 | 82 |
| minimal | flash | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.967 | fail | $0.9677 | 1 | 87 |
| headline | gemini_pro | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.713 | fail | $1.3438 | 1 | 46 |
| headline | gemini_pro | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.903 | fail | $0.7540 | 1 | 29 |
| headline | gemini_pro | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.979 | pass | $0.8793 | 1 | 40 |
| contract_only | haiku | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.956 | fail | $0.0647 | 1 | 54 |
| headline | haiku | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.951 | fail | $0.0917 | 1 | 52 |
| headline | haiku | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.802 | fail | $0.1937 | 1 | 59 |
| headline | haiku | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.981 | pass | $0.0451 | 1 | 37 |
| minimal | haiku | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.971 | fail | $0.0536 | 1 | 44 |
| headline | luna | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.872 | fail | $0.0125 | 1 | 19 |
| headline | luna | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.631 | fail | $0.0162 | 1 | 14 |
| headline | luna | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.922 | fail | $0.0174 | 1 | 17 |
| headline | nemotron | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.974 | fail | $0.1601 | 1 | 45 |
| headline | nemotron | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.644 | fail | $0.3024 | 1 | 85 |
| headline | nemotron | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.849 | fail | $0.2311 | 1 | 69 |
| contract_only | opus | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $1.8307 | 1 | 45 |
| headline | opus | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.997 | pass | $0.6830 | 1 | 20 |
| headline | opus | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.997 | pass | $0.5234 | 1 | 15 |
| headline | opus | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.999 | pass | $0.7654 | 1 | 21 |
| minimal | opus | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $0.9836 | 1 | 31 |
| contract_only | sol | t5_traces | frozen_and_off_target | trial 0 | submitted | 1 | pass | $0.2979 | 1 | 25 |
| headline | sol | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $0.3234 | 1 | 29 |
| headline | sol | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.999 | pass | $0.2392 | 1 | 19 |
| headline | sol | t5_traces | frozen_and_off_target | trial 2 | submitted | 1 | pass | $0.3247 | 1 | 28 |
| minimal | sol | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.994 | pass | $0.3356 | 1 | 29 |
| headline | sonnet | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.991 | pass | $0.4194 | 1 | 24 |
| headline | sonnet | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.99 | fail | $0.3258 | 1 | 23 |
| headline | sonnet | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.981 | pass | $0.4709 | 1 | 27 |
| contract_only | terra | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $0.1362 | 1 | 22 |
| headline | terra | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.998 | pass | $0.1809 | 1 | 30 |
| headline | terra | t5_traces | frozen_and_off_target | trial 1 | submitted | 0.998 | pass | $0.1275 | 1 | 20 |
| headline | terra | t5_traces | frozen_and_off_target | trial 2 | submitted | 0.997 | pass | $0.1760 | 1 | 27 |
| minimal | terra | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.998 | pass | $0.1884 | 1 | 29 |