Leaderboard
Every learner the paper ran, scored by the share of the feedback budget it used, U, where lower is better, beside the share of instances it solved within the cap, P. Each edition of ARC is its own track, and no score pools them.
Budget against spend
Each point is one learner at one effort on the workstreams stream: the share of the budget it used against what its model calls cost per instance, joined from the least effort to the most. ARC Prize plots score against cost per task the same way. Lower and further left is better.
Hover a point for its numbers, and click it to replay the run. A hollow point is a harness with everything it wrote deleted before every instance.
Every learner at medium effort
| Workstreams | Epilogue | Fresh | Sequential | Random | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | Learner | |||||||||||
| 1 | Program library | 11.7 ±4.2 | 93.5 | 22.0 ±4.0 | 88.0 | 20.5 ±3.6 | 88.0 | 10.1 ±3.6 | 95.5 | 8.9 ±3.4 | 95.5 | $0.005 |
| 2 | Prime Agentas shipped | 23.6 ±2.8 | 94.5 | 24.5 ±4.9 | 80.0 | 22.2 ±4.0 | 88.0 | 24.0 ±3.5 | 94.0 | 24.0 ±3.3 | 93.5 | $0.009 |
| 3 | OpenClawmemory wiped | 25.0 ±3.3 | 91.0 | 27.8 ±3.8 | 92.0 | 22.8 ±3.6 | 92.0 | 22.3 ±2.8 | 92.5 | 24.4 ±3.1 | 89.0 | $0.026 |
| 4 | Prime Agentmemory wiped | 25.1 ±3.5 | 92.0 | 25.0 ±4.1 | 84.0 | 17.8 ±2.8 | 100.0 | 23.7 ±3.5 | 93.0 | 23.0 ±2.9 | 94.5 | $0.009 |
| 5 | Demo-cache floor | 25.3 ±4.7 | 81.0 | 27.5 ±4.7 | 80.0 | 32.2 ±5.5 | 80.0 | 23.6 ±4.1 | 83.5 | 27.4 ±5.4 | 78.5 | $0.010 |
| 6 | OpenClawas shipped | 26.8 ±2.9 | 89.0 | 28.0 ±4.5 | 88.0 | 22.0 ±3.5 | 92.0 | 24.4 ±3.2 | 91.5 | 26.7 ±3.5 | 87.5 | $0.034 |
| 7 | Hermes Agentmemory wiped | 28.0 ±3.8 | 87.5 | 21.8 ±4.2 | 84.0 | 29.2 ±4.9 | 84.0 | 25.7 ±3.7 | 90.0 | 26.0 ±3.6 | 91.5 | $0.067 |
| 8 | Hermes Agentas shipped | 29.7 ±3.8 | 89.0 | 24.8 ±3.9 | 92.0 | 26.2 ±4.2 | 92.0 | 25.6 ±3.7 | 89.5 | 26.8 ±3.7 | 91.0 | $0.028 |
| 9 | Stateless floor | 33.3 ±4.0 | 83.0 | 31.0 ±4.3 | 84.0 | 27.5 ±4.0 | 84.0 | 32.8 ±4.1 | 83.5 | 32.6 ±4.8 | 81.0 | $0.009 |
U is the budget used and P the share solved within the cap, both in percent, over the stream's 200 instances or the 25 held-out ones. Small type is the standard error over tasks, as in the paper's tables. Epilogue is the held-out tasks after the workstreams stream; fresh is the same instances with the learner reset before each, the reference for transfer. Click a column to sort by it, and a number to replay its run.
Harnesses asked to store, or told to keep a program library
These rows change what a harness is told, so the paper reports them apart, in its Appendix F. Asked to store, a harness hears after every verdict that it should save anything that could help with the task's next instances. Told to keep a program library, it carries out the program library's method with its own tools, keeping one tested program per task and running it first when the task returns; wiped, it does the same with everything deleted before every instance. The last row is the program library with nothing kept. ARC-AGI-1 only.
| Workstreams | Epilogue | Sequential | Random | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| # | Learner | |||||||||
| 1 | Prime Agenttold to keep a program library | 12.3 | 96.5 | 25.5 | 92.0 | 8.2 | 99.0 | 9.9 | 97.5 | $0.010 |
| 2 | OpenClawtold to keep a program library | 14.2 | 93.5 | 26.5 | 84.0 | 15.5 | 93.0 | 12.3 | 97.5 | $0.060 |
| 3 | Hermes Agenttold to keep a program library | 15.0 | 93.0 | 24.8 | 88.0 | 13.8 | 94.0 | 14.8 | 94.0 | $0.071 |
| 4 | Prime Agentasked to store | 18.6 | 96.0 | 22.8 | 92.0 | 19.4 | 95.0 | 21.3 | 96.0 | $0.012 |
| 5 | Hermes Agentasked to store | 20.7 | 89.5 | 33.8 | 72.0 | 22.7 | 89.5 | 17.1 | 92.5 | $0.031 |
| 6 | OpenClawtold to keep a program library, memory wiped | 23.1 | 94.5 | 22.5 | 96.0 | 22.8 | 93.5 | 23.0 | 95.0 | $0.089 |
| 7 | Prime Agenttold to keep a program library, memory wiped | 23.4 | 94.5 | 23.5 | 88.0 | 22.3 | 96.0 | 20.2 | 97.0 | $0.014 |
| 8 | Program library, nothing kept | 23.7 | 90.5 | 22.0 | 88.0 | 24.8 | 92.0 | 23.2 | 91.5 | $0.007 |
| 9 | Hermes Agenttold to keep a program library, memory wiped | 26.6 | 89.0 | 21.5 | 96.0 | 26.9 | 88.5 | 27.1 | 88.5 | $0.11 |
| 10 | OpenClawasked to store | 28.0 | 87.5 | 27.0 | 84.0 | 26.0 | 88.0 | 24.5 | 90.5 | $0.031 |
The paper reads these rows against each other, instance by instance, rather than as absolute numbers, so they carry no error bar here.
Reported results
Results from outside unify, run on the public stream and sent with their run records, go here. Each is marked reported until we rerun it from its code, and then verified.