130 trials · total cost $27.97 · 2 Sol T3 episodes ended on an OpenAI rate limit (shown as harness failures; fails in the all-pass counts) · not on this page: 100 attempts that ended on an empty OpenAI account ($0.23, re-run until completed); they are in the dataset's experiments/sweep1_openai/setaside/ · click a trial to step through it
| agent | task | trials | harness errors | submitted | all pass | mean end score | cost | cost / trial |
|---|---|---|---|---|---|---|---|---|
| sol | t1_connectivity | 14 | — | 14 | 9 / 14 | 0.84 | $4.4179 | $0.3156 |
| sol | t2_setup | 15 | — | 15 | 9 / 15 | 0.964 | $1.6137 | $0.1076 |
| sol | t3_light | 16 | 2 | 14 | 11 / 16 | 0.923 | $6.2606 | $0.3913 |
| sol | t4_takeover | 11 | — | 11 | 10 / 11 | 0.909 | $3.2095 | $0.2918 |
| sol | t5_traces | 9 | — | 9 | 9 / 9 | 0.998 | $2.4967 | $0.2774 |
| sol | all tasks | 65 | 2 | 63 | 48 / 65 | 0.923 | $17.9984 | $0.2769 |
| terra | t1_connectivity | 14 | — | 14 | 10 / 14 | 0.984 | $2.2905 | $0.1636 |
| terra | t2_setup | 15 | — | 15 | 8 / 15 | 0.951 | $1.0392 | $0.0693 |
| terra | t3_light | 16 | — | 16 | 12 / 16 | 0.935 | $3.0514 | $0.1907 |
| terra | t4_takeover | 11 | — | 11 | 6 / 11 | 0.905 | $2.0538 | $0.1867 |
| terra | t5_traces | 9 | — | 9 | 8 / 9 | 0.98 | $1.5364 | $0.1707 |
| terra | all tasks | 65 | — | 65 | 44 / 65 | 0.951 | $9.9713 | $0.1534 |
Each trial has one end grade, computed once after the run ends. Harness failures (the run stopped on a provider or harness error) are listed but have no score: they are left out of "all pass" and the means.
| headline | sol | t1_connectivity | dead_culture | trial 0 | submitted | 1 | pass | $0.3969 | 45 | 71 |
| headline | sol | t1_connectivity | hdmi_loose | trial 0 | submitted | 1 | pass | $0.2517 | 27 | 45 |
| headline | sol | t1_connectivity | led_power_loose | trial 0 | submitted | 0.846 | fail | $0.2420 | 28 | 44 |
| headline | sol | t1_connectivity | loose_wire | trial 0 | submitted | 1 | pass | $0.3825 | 37 | 66 |
| headline | sol | t1_connectivity | low_disk | trial 0 | submitted | 0 | fail | $0.1618 | 18 | 27 |
| headline | sol | t1_connectivity | mask_flip | trial 0 | submitted | 1 | pass | $0.3878 | 46 | 68 |
| headline | sol | t1_connectivity | none | trial 0 | submitted | 1 | pass | $0.3382 | 36 | 58 |
| headline | sol | t1_connectivity | operator_away | trial 0 | submitted | 1 | pass | $0.2920 | 35 | 50 |
| headline | sol | t1_connectivity | projector_standby | trial 0 | submitted | 1 | pass | $0.3206 | 30 | 55 |
| headline | sol | t1_connectivity | red_led_aged | trial 0 | submitted | 0.957 | fail | $0.4063 | 37 | 59 |
| headline | sol | t1_connectivity | room_light | trial 0 | submitted | 0 | fail | $0.3612 | 48 | 65 |
| headline | sol | t1_connectivity | stale_frames | trial 0 | submitted | 1 | pass | $0.2756 | 30 | 49 |
| headline | sol | t1_connectivity | trigger_cable | trial 0 | submitted | 0.96 | fail | $0.3068 | 33 | 52 |
| headline | sol | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.2945 | 30 | 48 |
| headline | terra | t1_connectivity | dead_culture | trial 0 | submitted | 0.883 | fail | $0.2102 | 36 | 61 |
| headline | terra | t1_connectivity | hdmi_loose | trial 0 | submitted | 1 | pass | $0.1700 | 19 | 34 |
| headline | terra | t1_connectivity | led_power_loose | trial 0 | submitted | 1 | pass | $0.1231 | 27 | 42 |
| headline | terra | t1_connectivity | loose_wire | trial 0 | submitted | 1 | pass | $0.1930 | 32 | 53 |
| headline | terra | t1_connectivity | low_disk | trial 0 | submitted | 1 | pass | $0.1313 | 24 | 39 |
| headline | terra | t1_connectivity | mask_flip | trial 0 | submitted | 0.969 | fail | $0.1717 | 38 | 55 |
| headline | terra | t1_connectivity | none | trial 0 | submitted | 1 | pass | $0.1262 | 19 | 39 |
| headline | terra | t1_connectivity | operator_away | trial 0 | submitted | 1 | pass | $0.1276 | 21 | 37 |
| headline | terra | t1_connectivity | projector_standby | trial 0 | submitted | 1 | pass | $0.1340 | 25 | 42 |
| headline | terra | t1_connectivity | red_led_aged | trial 0 | submitted | 0.969 | fail | $0.1706 | 24 | 48 |
| headline | terra | t1_connectivity | room_light | trial 0 | submitted | 1 | pass | $0.2241 | 33 | 52 |
| headline | terra | t1_connectivity | stale_frames | trial 0 | submitted | 1 | pass | $0.2215 | 38 | 62 |
| headline | terra | t1_connectivity | trigger_cable | trial 0 | submitted | 0.96 | fail | $0.1329 | 26 | 43 |
| headline | terra | t1_connectivity | usb_loose | trial 0 | submitted | 1 | pass | $0.1543 | 25 | 45 |
| headline | sol | t2_setup | calibration_missing | trial 0 | submitted | 0.956 | fail | $0.0893 | 15 | 22 |
| headline | sol | t2_setup | calibration_stale | trial 0 | submitted | 0.867 | fail | $0.0924 | 16 | 22 |
| headline | sol | t2_setup | dense_culture | trial 0 | submitted | 1 | pass | $0.0973 | 15 | 23 |
| headline | sol | t2_setup | distortion_dense | trial 0 | submitted | 0.867 | fail | $0.0921 | 14 | 20 |
| headline | sol | t2_setup | few_markers | trial 0 | submitted | 1 | pass | $0.2191 | 24 | 33 |
| headline | sol | t2_setup | hdmi_loose | trial 0 | submitted | 1 | pass | $0.0707 | 12 | 18 |
| headline | sol | t2_setup | lens_distortion | trial 0 | submitted | 1 | pass | $0.1111 | 17 | 25 |
| headline | sol | t2_setup | loose_wire | trial 0 | submitted | 1 | pass | $0.1010 | 20 | 27 |
| headline | sol | t2_setup | mask_flip | trial 0 | submitted | 0.956 | fail | $0.0899 | 13 | 20 |
| headline | sol | t2_setup | missing_uneven | trial 0 | submitted | 0.956 | fail | $0.1520 | 19 | 29 |
| headline | sol | t2_setup | none | trial 0 | submitted | 1 | pass | $0.0775 | 13 | 17 |
| headline | sol | t2_setup | operator_away | trial 0 | submitted | 1 | pass | $0.0918 | 18 | 25 |
| headline | sol | t2_setup | projector_standby | trial 0 | submitted | 1 | pass | $0.1194 | 19 | 28 |
| headline | sol | t2_setup | room_light | trial 0 | submitted | 0.867 | fail | $0.0879 | 15 | 20 |
| headline | sol | t2_setup | uneven_illumination | trial 0 | submitted | 1 | pass | $0.1224 | 14 | 25 |
| headline | terra | t2_setup | calibration_missing | trial 0 | submitted | 0.956 | fail | $0.0451 | 12 | 17 |
| headline | terra | t2_setup | calibration_stale | trial 0 | submitted | 1 | pass | $0.0666 | 15 | 23 |
| headline | terra | t2_setup | dense_culture | trial 0 | submitted | 1 | pass | $0.0852 | 17 | 25 |
| headline | terra | t2_setup | distortion_dense | trial 0 | submitted | 0.867 | fail | $0.0697 | 16 | 25 |
| headline | terra | t2_setup | few_markers | trial 0 | submitted | 0.956 | fail | $0.1263 | 25 | 36 |
| headline | terra | t2_setup | hdmi_loose | trial 0 | submitted | 1 | pass | $0.0489 | 14 | 20 |
| headline | terra | t2_setup | lens_distortion | trial 0 | submitted | 0.867 | fail | $0.0538 | 14 | 21 |
| headline | terra | t2_setup | loose_wire | trial 0 | submitted | 1 | pass | $0.0604 | 16 | 24 |
| headline | terra | t2_setup | mask_flip | trial 0 | submitted | 1 | pass | $0.0625 | 14 | 20 |
| headline | terra | t2_setup | missing_uneven | trial 0 | submitted | 0.948 | fail | $0.1132 | 19 | 32 |
| headline | terra | t2_setup | none | trial 0 | submitted | 1 | pass | $0.0566 | 15 | 20 |
| headline | terra | t2_setup | operator_away | trial 0 | submitted | 1 | pass | $0.0509 | 15 | 21 |
| headline | terra | t2_setup | projector_standby | trial 0 | submitted | 1 | pass | $0.0613 | 19 | 24 |
| headline | terra | t2_setup | room_light | trial 0 | submitted | 0.867 | fail | $0.0398 | 15 | 19 |
| headline | terra | t2_setup | uneven_illumination | trial 0 | submitted | 0.811 | fail | $0.0989 | 19 | 29 |
| headline | sol | t3_light | close_neighbours | trial 0 | submitted | 0.977 | pass | $0.3030 | 37 | 68 |
| headline | sol | t3_light | cluster_fast_bleach | trial 0 | submitted | 0.986 | pass | $0.3095 | 37 | 59 |
| headline | sol | t3_light | converging | trial 0 | harness error | no score | — | $0.7471 | 54 | 101 |
| headline | sol | t3_light | converging_led_low | trial 0 | submitted | 0.973 | pass | $0.5479 | 59 | 93 |
| headline | sol | t3_light | crosstalk_two_targets | trial 0 | submitted | 0.828 | fail | $0.2775 | 36 | 56 |
| headline | sol | t3_light | dead_culture | trial 0 | submitted | 0.976 | pass | $0.3630 | 50 | 78 |
| headline | sol | t3_light | fast_bleach | trial 0 | submitted | 0.667 | fail | $0.3051 | 42 | 64 |
| headline | sol | t3_light | led_low | trial 0 | submitted | 0.981 | pass | $0.3477 | 39 | 69 |
| headline | sol | t3_light | loose_wire | trial 0 | submitted | 0.968 | pass | $0.3380 | 38 | 66 |
| headline | sol | t3_light | none | trial 0 | submitted | 1 | pass | $0.2488 | 32 | 52 |
| headline | sol | t3_light | red_led_aged | trial 0 | submitted | 0.986 | pass | $0.3734 | 39 | 71 |
| headline | sol | t3_light | room_light | trial 0 | submitted | 0.667 | fail | $0.3872 | 39 | 64 |
| headline | sol | t3_light | stale_frames | trial 0 | harness error | no score | — | $0.4888 | 45 | 76 |
| headline | sol | t3_light | strong_crosstalk | trial 0 | submitted | 0.992 | pass | $0.2706 | 34 | 56 |
| headline | sol | t3_light | strong_input | trial 0 | submitted | 0.94 | pass | $0.3044 | 39 | 64 |
| headline | sol | t3_light | two_targets_room_light | trial 0 | submitted | 0.98 | pass | $0.6487 | 60 | 103 |
| headline | terra | t3_light | close_neighbours | trial 0 | submitted | 0.998 | pass | $0.1269 | 29 | 49 |
| headline | terra | t3_light | cluster_fast_bleach | trial 0 | submitted | 0.998 | pass | $0.1630 | 31 | 56 |
| headline | terra | t3_light | converging | trial 0 | submitted | 0.943 | pass | $0.3581 | 85 | 127 |
| headline | terra | t3_light | converging_led_low | trial 0 | submitted | 0.667 | fail | $0.3755 | 76 | 112 |
| headline | terra | t3_light | crosstalk_two_targets | trial 0 | submitted | 0.72 | fail | $0.1712 | 32 | 55 |
| headline | terra | t3_light | dead_culture | trial 0 | submitted | 0.975 | pass | $0.1410 | 38 | 59 |
| headline | terra | t3_light | fast_bleach | trial 0 | submitted | 0.991 | pass | $0.1306 | 29 | 50 |
| headline | terra | t3_light | led_low | trial 0 | submitted | 0.898 | fail | $0.1619 | 31 | 57 |
| headline | terra | t3_light | loose_wire | trial 0 | submitted | 0.984 | pass | $0.1451 | 34 | 50 |
| headline | terra | t3_light | none | trial 0 | submitted | 0.981 | pass | $0.1294 | 27 | 46 |
| headline | terra | t3_light | red_led_aged | trial 0 | submitted | 0.959 | pass | $0.1747 | 35 | 63 |
| headline | terra | t3_light | room_light | trial 0 | submitted | 0.967 | fail | $0.1827 | 33 | 59 |
| headline | terra | t3_light | stale_frames | trial 0 | submitted | 0.969 | pass | $0.2020 | 35 | 65 |
| headline | terra | t3_light | strong_crosstalk | trial 0 | submitted | 0.976 | pass | $0.1853 | 28 | 55 |
| headline | terra | t3_light | strong_input | trial 0 | submitted | 0.953 | pass | $0.1473 | 31 | 54 |
| headline | terra | t3_light | two_targets_room_light | trial 0 | submitted | 0.977 | pass | $0.2567 | 41 | 76 |
| headline | sol | t4_takeover | clean | trial 0 | submitted | 1 | pass | $0.2703 | 22 | 40 |
| headline | sol | t4_takeover | dead_culture_swap | trial 0 | submitted | 1 | pass | $0.3158 | 27 | 48 |
| headline | sol | t4_takeover | disk_note_fine | trial 0 | submitted | 0 | fail | $0.2304 | 24 | 40 |
| headline | sol | t4_takeover | disk_triage | trial 0 | submitted | 1 | pass | $0.2437 | 23 | 37 |
| headline | sol | t4_takeover | flip_grid_check | trial 0 | submitted | 1 | pass | $0.3729 | 39 | 63 |
| headline | sol | t4_takeover | frozen_counters | trial 0 | submitted | 1 | pass | $0.2778 | 25 | 42 |
| headline | sol | t4_takeover | loose_jumper | trial 0 | submitted | 1 | pass | $0.2124 | 14 | 29 |
| headline | sol | t4_takeover | moved_sample | trial 0 | submitted | 1 | pass | $0.3245 | 23 | 46 |
| headline | sol | t4_takeover | t_dead_jumper | trial 0 | submitted | 1 | pass | $0.2639 | 19 | 39 |
| headline | sol | t4_takeover | t_moved_frozen | trial 0 | submitted | 1 | pass | $0.3121 | 26 | 46 |
| headline | sol | t4_takeover | t_present_faults | trial 0 | submitted | 1 | pass | $0.3856 | 36 | 62 |
| headline | terra | t4_takeover | clean | trial 0 | submitted | 1 | pass | $0.1334 | 14 | 27 |
| headline | terra | t4_takeover | dead_culture_swap | trial 0 | submitted | 1 | pass | $0.3295 | 40 | 76 |
| headline | terra | t4_takeover | disk_note_fine | trial 0 | submitted | 1 | pass | $0.1249 | 17 | 32 |
| headline | terra | t4_takeover | disk_triage | trial 0 | submitted | 0.998 | fail | $0.1662 | 22 | 39 |
| headline | terra | t4_takeover | flip_grid_check | trial 0 | submitted | 1 | pass | $0.1806 | 31 | 50 |
| headline | terra | t4_takeover | frozen_counters | trial 0 | submitted | 1 | pass | $0.1466 | 21 | 37 |
| headline | terra | t4_takeover | loose_jumper | trial 0 | submitted | 0.971 | fail | $0.1365 | 16 | 31 |
| headline | terra | t4_takeover | moved_sample | trial 0 | submitted | 1 | pass | $0.1714 | 13 | 34 |
| headline | terra | t4_takeover | t_dead_jumper | trial 0 | submitted | 0.988 | fail | $0.1551 | 17 | 38 |
| headline | terra | t4_takeover | t_moved_frozen | trial 0 | submitted | 0.996 | fail | $0.2854 | 23 | 54 |
| headline | terra | t4_takeover | t_present_faults | trial 0 | submitted | 0 | fail | $0.2240 | 27 | 47 |
| headline | sol | t5_traces | bleached | trial 0 | submitted | 0.999 | pass | $0.3213 | 1 | 31 |
| headline | sol | t5_traces | bleached_and_dropped | trial 0 | submitted | 0.998 | pass | $0.3091 | 1 | 25 |
| headline | sol | t5_traces | clean | trial 0 | submitted | 0.998 | pass | $0.2602 | 1 | 24 |
| headline | sol | t5_traces | dropped_and_light | trial 0 | submitted | 0.997 | pass | $0.2590 | 1 | 22 |
| headline | sol | t5_traces | everything | trial 0 | submitted | 0.997 | pass | $0.3250 | 1 | 26 |
| headline | sol | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.999 | pass | $0.2714 | 1 | 22 |
| headline | sol | t5_traces | frozen_stretch | trial 0 | submitted | 1 | pass | $0.2783 | 1 | 26 |
| headline | sol | t5_traces | light_and_frozen | trial 0 | submitted | 0.999 | pass | $0.2715 | 1 | 24 |
| headline | sol | t5_traces | off_target | trial 0 | submitted | 0.998 | pass | $0.2010 | 1 | 17 |
| headline | terra | t5_traces | bleached | trial 0 | submitted | 0.847 | fail | $0.1739 | 1 | 25 |
| headline | terra | t5_traces | bleached_and_dropped | trial 0 | submitted | 0.993 | pass | $0.2199 | 1 | 33 |
| headline | terra | t5_traces | clean | trial 0 | submitted | 0.999 | pass | $0.1496 | 1 | 20 |
| headline | terra | t5_traces | dropped_and_light | trial 0 | submitted | 0.995 | pass | $0.1782 | 1 | 25 |
| headline | terra | t5_traces | everything | trial 0 | submitted | 0.998 | pass | $0.2003 | 1 | 27 |
| headline | terra | t5_traces | frozen_and_off_target | trial 0 | submitted | 0.997 | pass | $0.1803 | 1 | 26 |
| headline | terra | t5_traces | frozen_stretch | trial 0 | submitted | 0.999 | pass | $0.1813 | 1 | 32 |
| headline | terra | t5_traces | light_and_frozen | trial 0 | submitted | 0.999 | pass | $0.1242 | 1 | 16 |
| headline | terra | t5_traces | off_target | trial 0 | submitted | 0.997 | pass | $0.1286 | 1 | 25 |