A continual learning benchmark · 3 tracks · 257 runs

Continual-ARC

A lifelong stream of ARC puzzles that come back with fresh inputs after long gaps, while new ones keep arriving. It measures whether a learner gets cheaper on work it has seen before, whether it keeps notes like an agent harness or updates its weights.

Results · ARC-AGI-1

Budget used each time a task comes back

We ran three agent harnesses through the stream with every learning mechanism they ship switched on, then again with everything they wrote deleted before every instance. A learner that has learned a task should fall below the dotted line, where the demo-cache floor sits once it holds the task's demonstrations. The harnesses start a few points under the bare model, since they can test a guess against the demonstrations, and stay there however often the task returns. The program library, a deliberately naive baseline built around ARC, falls well below.

GPT-6 Luna · workstreams

demo-cache floor after its first meeting, 24.9

Replay a run at medium effort:Stateless floor Demo-cache floor Program library Prime Agent, as shipped Hermes Agent, as shipped OpenClaw, as shipped

Saved by the harnesses' memory: −6.1 to +1.0points; Each harness against itself wiped before every instance, at every effort. None saved measurably more than nothing.
Returning tasks solved for free: 1.3percent at most; Left to decide what to keep, no harness answered more of its returning instances right first time without a demonstration. The program library: 68 to 87 percent.
The program library's lead: 13 to 30points less; Budget it used below every harness, at every effort. No harness measurably beat even the demo-cache floor.

The curves are the paper's Figure 3, on the workstreams stream. The three numbers pair each comparison instance by instance and pool the three orders, with 95 percent intervals by bootstrap over tasks; the returning tasks are those of Appendix F. Every figure of the paper is on Figures.

The benchmark

How it works

One instance

The learner sees the task's id and a fresh input grid. It can buy a demonstration, one more input with its output, or submit an answer and hear only right or wrong. A demonstration and a wrong answer each cost 1, up to eight of each, and a right answer ends the instance.

Step through a real one, or play it.

Test input

Demonstrations bought

None yet

Answers

None yet

DemonstrationsWrong answers0 of 16 spent

Hermes Agent meets task 46442a0e for the first time: the task's id and a test input, nothing more.

Step 21 of the ARC-AGI-1 workstreams stream, as Hermes Agent played it on GPT-6 Luna, medium. Open it in the replay for everything it reasoned and called.

A stream of instances

25 tasks over 200 instances, four active at a time, each worked in bursts of about three and resting about thirty, with a new task every two. A task came back after a median gap of 5 instances, and at most 141. Then 25 held-out tasks, once each.

The task played above is lit in it.

Point at a column to see which task it served; the task’s marks light up.

first meeting with the taskthe task comes backtask active, in a spellheld-out task, served once

The score

An instance is charged its demonstrations and wrong answers out of 16. U is the share of that budget used, averaged over the stream, so lower is better, and P, the share solved within the cap, sits beside it. A learner that has kept a task's rule can answer at once and spend nothing.

One that has learned the task answers it at once, so its line falls towards zero. In the runs the nearest is the program library: U 11.7 and P 93.5 on the workstreams stream, GPT-6 Luna, medium.

ARC-AGI-125 of the 400 training tasks, every instance fresh from Re-ARC's generators, then 25 held-out evaluation tasks.
ARC-AGI-225 of the 114 evaluation tasks new to the second edition, each visit under a symmetry and a recolouring its rule survives, then 25 more held out.
ARC-AGI-3Six games over 48 levels, then four held-out games, charged in actions against the human baseline.

Each edition gets its own track and its own number, and no score pools them. The tasks of every track are on Tasks.

Leaderboard · ARC-AGI-1

Budget used on the workstreams stream

The full leaderboard

#	Learner	GPT-6 Luna, low	GPT-6 Luna, medium	GPT-6 Luna, high
1	Program library	15.7 ±4.2	11.7 ±4.2	7.8 ±3.4
2	Prime Agentas shipped	42.8 ±4.7	23.6 ±2.8	20.2 ±2.5
3	OpenClawmemory wiped	39.3 ±4.0	25.0 ±3.3	23.2 ±2.9
4	Prime Agentmemory wiped	42.1 ±4.5	25.1 ±3.5	20.5 ±2.3
5	Demo-cache floor	38.1 ±5.2	25.3 ±4.7	17.4 ±3.8
6	OpenClawas shipped	42.3 ±4.2	26.8 ±2.9	23.2 ±2.8
7	Hermes Agentmemory wiped	39.0 ±4.4	28.0 ±3.8	23.6 ±3.4
8	Hermes Agentas shipped	44.2 ±4.9	29.7 ±3.8	21.6 ±2.6
9	Stateless floor	49.0 ±5.2	33.3 ±4.0	26.9 ±3.3

U in percent of the budget, lower is better, with the standard error over tasks; ranked at medium effort. Each number opens the replay of its run.

Replay · GPT-6 Luna, medium · workstreams

Watch a run

Every instance of every run can be replayed, from what the learner bought and submitted to what a harness chose to keep after the verdict. Each pair below ran on the same schedule, so their instances line up one for one.

OpenClaw, as shipped and asked to store

As shipped, OpenClaw wrote no memory. Asked after every verdict to save anything that could help, it wrote notes under names it chose afresh each time, and before acting almost never looked for them when the task came back.

Prime Agent, as shipped and asked to store

Prime Agent's lessons live in the session unless the agent marks one global, and as shipped it never did. Asked to store, its global entries opened every later session, and returning tasks got cheaper.

Hermes Agent, as shipped and told to keep programs

Hermes Agent's background review wrote skills of general advice, many named for steps of the protocol. Told to keep one tested program per task and run it first, it carried the method out with its own tools.

The program library, kept and wiped

Every answer is a Python program, checked free against the demonstrations. A program that reproduces them all is kept and runs first when its task returns. With its programs wiped before every instance, it writes each one again from scratch.

Or choose any run by track, learner, variant, effort and order on Replay.

Run your own

Run it on your learner

The stream and the records of every run are public, and there is no held-out set. Run your learner or harness on the canonical stream, send the run directory as a pull request, and it joins the leaderboard marked reported until we rerun it.

shell

git clone https://github.com/unifyai/continual-arc
cd continual-arc
uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch
uv run continual-arc schedule make --config arc1-small \
  --mode workstreams --seed 0 --out arc1-ws0.json

How to run a harness and submit a result

Cite

BibTeX

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime. Daniel Lenton, unify, 2026.

bibtex

@misc{lenton2026continualarc,
  title  = {Continual-ARC: Measuring Skill Acquisition over a Working Lifetime},
  author = {Lenton, Daniel},
  year   = {2026},
  url    = {https://unify.ai/continual-arc}
}