Set-aside attempts: attempts cut short by the harness or provider; each was re-run until it completed, and the completed run is the scored record. Most needed one re-run; argon T1 trial 1 needed two, and opus T3 trial 0 and fable T5 trial 0 needed three (one of fable's three is an aborted record kept in the dataset, not shown here). Trials set aside more than once carry -2, -3 ids; each page's footer names its source folder. Not part of the headline.

optobench runs

19 trials · 8 harness failures (no score) · total cost $25.21 · click a trial to step through it

agentscenariotaskcondition

By agent and task (trials shown by the filters)

agent	task	trials	harness errors	submitted	all pass	mean end score	cost	cost / trial
argon	t1_connectivity	3	—	0	0 / 3	0	$1.6176	$0.5392
argon	t4_takeover	1	—	0	0 / 1	0	$0.8973	$0.8973
argon	t5_traces	1	—	1	1 / 1	0.999	$2.4012	$2.4012
argon	all tasks	5	—	1	1 / 5	0.2	$4.9161	$0.9832
astra	t3_light	1	—	0	0 / 1	0	$5.0195	$5.0195
deepseek	t5_traces	1	—	1	1 / 1	0.993	$0.0857	$0.0857
fable	t3_light	1	1	0	0 / 0	—	$1.6442	$1.6442
fable	t5_traces	2	1	1	1 / 1	0.998	$5.1993	$2.5996
fable	all tasks	3	2	1	1 / 1	0.998	$6.8435	$2.2812
nemotron	t5_traces	2	—	2	0 / 2	0.734	$0.2696	$0.1348
opus	t3_light	3	2	1	0 / 1	0.818	$7.3560	$2.4520
sol	t1_connectivity	2	2	0	0 / 0	—	$0.4537	$0.2269
terra	t1_connectivity	1	1	0	0 / 0	—	$0.0917	$0.0917
terra	t5_traces	1	1	0	0 / 0	—	$0.1755	$0.1755
terra	all tasks	2	2	0	0 / 0	—	$0.2673	$0.1336

Each trial has one end grade, computed once after the run ends. Harness failures (the run stopped on a provider or harness error) are listed but have no score: they are left out of "all pass" and the means.

Trials


headline	argon	t1_connectivity	usb_loose	trial 0	refused	0	fail	$1.1340	34	71
headline	argon	t1_connectivity	usb_loose	trial 1	refused	0	fail	$0.3236	15	23
headline	argon	t1_connectivity	usb_loose	trial 1	refused	0	fail	$0.1599	8	15
headline	sol	t1_connectivity	mask_flip	trial 0	harness error	no score	—	$0.2959	45	64
headline	sol	t1_connectivity	stale_frames	trial 0	harness error	no score	—	$0.1578	21	32
headline	terra	t1_connectivity	red_led_aged	trial 0	harness error	no score	—	$0.0917	19	34
headline	astra	t3_light	two_targets_room_light	trial 0	no submission	0	fail	$5.0195	58	103
headline	fable	t3_light	two_targets_room_light	trial 1	harness error	no score	—	$1.6442	21	35
headline	opus	t3_light	two_targets_room_light	trial 0	harness error	no score	—	$3.4152	54	110
headline	opus	t3_light	two_targets_room_light	trial 0	harness error	no score	—	$1.1393	32	58
headline	opus	t3_light	two_targets_room_light	trial 0	submitted	0.818	fail	$2.8014	49	113
headline	argon	t4_takeover	t_present_faults	trial 0	refused	0	fail	$0.8973	16	44
headline	argon	t5_traces	frozen_and_off_target	trial 0	submitted	0.999	pass	$2.4012	1	68
headline	deepseek	t5_traces	frozen_and_off_target	trial 0	submitted	0.993	pass	$0.0857	1	100
headline	fable	t5_traces	frozen_and_off_target	trial 0	harness error	no score	—	$1.4708	0	14
headline	fable	t5_traces	frozen_and_off_target	trial 0	submitted	0.998	pass	$3.7285	1	27
headline	nemotron	t5_traces	frozen_and_off_target	trial 0	submitted	0.838	fail	$0.1396	1	57
headline	nemotron	t5_traces	frozen_and_off_target	trial 1	submitted	0.631	fail	$0.1300	1	50
headline	terra	t5_traces	frozen_and_off_target	trial 1	harness error	no score	—	$0.1755	0	24