Why we built Continual-ARC.

Agent harnesses such as OpenClaw[1], Hermes Agent[2] and Prime Agent[3] keep notes and write themselves skills, and they're often described as a way for an agent to keep learning on the job. We wanted to know what kind of learning that is, and to put it on the same scale as training the model's weights as it works, but couldn't find a benchmark that would let us.

Learning on the job can mean two things. Someone who has done the same client's quarterly accounts four times is quicker the fifth time, because they remember how that client's books work. Someone who has done a hundred clients' accounts is quicker with a new client too, because they've got better at accounts. The first is memory, and the second is the learning we care about most.

ARC measures how efficiently a system picks up a new skill from a few examples[4], but only once. The continual-learning benchmarks for agents[7][8][9][10][11][12][13] mostly sit in one domain, such as coding, where the model already knows a great deal and part of a method's gain may come from drawing that knowledge out[14]. Since they serve each task once, a remembered answer can't be told apart from a learned skill, and none checks afterwards whether the agent got better at tasks it never saw.

So we built Continual-ARC, a long stream of ARC puzzles in which the same puzzles keep returning with fresh inputs while new ones arrive. On every visit, the learner can pay to see a worked example or try an answer, and a wrong answer costs as much as an example. At the end we freeze the learner and give it two tests: fresh inputs of the puzzles it met, which measure memory, and 100 puzzles it never met, which measure whether it got better at ARC. Every learner lands on a plane of the two, and a harness keeping notes and a model updating its weights face exactly the same stream.

We put one open model, Qwen3.5-9B, through it both ways, with and without a stronger teacher, GLM-5.3 Flash. In one, Qwen keeps notes in Prime Agent, or GLM writes the notes for it. In the other, the same games train Qwen's weights, by imitating GLM's moves or by learning from the games it won. Left to itself, Qwen's harness saved nothing measurable, on the puzzles it had met or on new ones, and its notes did little better with our guidance or with GLM writing them. Trained to imitate GLM, Qwen's weights saved nearly half of what the untrained model spent on the puzzles it had met, and about a quarter on the 100 new ones, more than twice what a page of GLM's playing rules saved the untrained model.

Since a 9B model might simply keep poor notes, we also had GLM, which beats GPT-6 Luna on general benchmarks, play the same harness itself. Left to decide what to keep, it still paid for most puzzles it had met, and given our guidance, for under a third, as did three harnesses on GPT-6 Luna. We think the gap is in what the harness chooses to keep rather than in the model.

Two kinds of learning, two tests

An ARC puzzle gives you a few example grids, each an input and its output, and you work out the rule that produces the output for one more input. The puzzles need only core priors like objectness and counting, meaning a learner's gains should depend less on what the model already knows. However, ARC never serves a puzzle twice, and the strongest ARC-AGI-1 systems adapt their weights at test time and then throw the adaptation away[5].

Michael Hodel's Re-ARC[6] has a generator and a verifier for each of ARC-AGI-1's 400 training puzzles, and with these we can bring a puzzle back as often as we like, each time with a fresh, verified input. A remembered answer never fits the new grid, and what's worth carrying from one visit to the next is the rule, not the answer.

We serve the puzzles on a schedule shaped like an office worker's week, where the report you wrote in March is due again in June. Four puzzles are active at a time, each for about three visits and then a rest of about thirty, until 25 puzzles have had eight visits each. The last new puzzle arrives in round 97 of 200.

We then freeze the learner and give it two tests of 100 visits each, played at once from where the stream left it. The seen test brings back the stream's 25 puzzles with fresh inputs, and the held-out test is 100 puzzles from ARC-AGI-1's evaluation set that the stream never served. A learner that keeps one solution per puzzle should get cheaper on the first and no cheaper on the second, and one that has got better at ARC should get cheaper on both.

The same design runs on ARC-AGI-2[21], with Google's ARC-GEN generators[22], and on ARC-AGI-3's games[23]. Every experiment here is on ARC-AGI-1.

How a visit is scored

A worked example is one more input and its output for the same puzzle, and a visit can ask for up to eight. A visit ends on a right answer or the eighth wrong one, and a failed visit costs the whole budget of sixteen, which makes solving always cheaper than failing. We report the share of the budget used, where lower is better.

Without a price on examples, a system with no memory would do fine and the benchmark would be ARC again. With one, a system that has learned a puzzle pays nothing, and one that hasn't pays to look.

A raw score would mostly measure the model, though, since a stronger model needs fewer examples whether or not it remembers anything. We read each learner against itself reset instead, with everything it kept wiped before every visit. On each test, we give the share of its reset's spend that it no longer pays, where 0 is no change and 1 is the whole test for free. The harness counts as part of the baseline, because it's where a harness-based learner does its learning. Each learner and its reset play each test four times, and we count a gain only where its 95 percent interval leaves out zero.

We show the puzzle's name on every visit, since recognizing a puzzle is a separate skill from remembering how to solve it.

Every learner on one plane

Each learner lands on one plane: across, the share of its reset's spend it saved on the puzzles it had met, which is memory, and up, the share it saved on the 100 it never met, which is getting better at ARC. The top right is better on both, and a result is a learning method on a chosen student model and, if it has one, a teacher.

A share can flatter, though. A learner whose reset already spends little can save a large share of very little, which is why we read the plane beside the budget each learner actually used.

The student and teacher are published models used unchanged, and a result is compared only with others on the same pair: a model made to fail at first, a few training steps from its usual self, would otherwise save a large share by recovering alone.

Notes against weights, with one teacher

We use a simple baseline, the program library, which runs the model with no harness. It keeps the examples it has paid for on each puzzle, and every answer has to come from a Python function from grid to grid that the model writes, checked free against them. A program that reproduces them all is kept and run first when the puzzle returns. It assumes, as only ARC allows, that every rule is a function of the grid.

We run every harness on Qwen and GLM in Prime Agent, with its own refinement switched on. After each verdict the harness gets a turn to keep whatever it likes, and as shipped, Qwen chooses what that is. We wrote our guidance as a domain expert would write a procedure, here the program library's: keep each puzzle's examples and a checked program in files, and leave a note in memory saying where they are. Qwen carries it out with its own tools. In the teacher's versions, GLM takes the harness's turn after each verdict: it reads Qwen's whole game and keeps whatever it thinks will help, while Qwen plays every move.

The trained versions give the same teaching to Qwen's weights. Between rounds, we train a small adapter[16] on every game so far. One imitates GLM's move in each position Qwen met[17], with GLM reading a page of playing rules that Qwen never sees. One learns from the games it won, with no teacher. A third distills Qwen into itself, reading GLM's move as an example[18].

What the notes and the weights remembered

On the stream, Qwen's harness as shipped learned nothing measurable, and neither our guidance nor GLM's notes raised that measurably. On the seen test, only one saved more than chance: 0.12, when GLM wrote the notes and both were given our guidance. With Qwen keeping them, our guidance saved 0.07, just short.

Trained to imitate GLM, Qwen's weights saved 0.45 of the untrained model's spend on the seen test, taking it from 87 to 48 percent of the budget. Learning from the games it won saved 0.07 there, and self-distillation did worse than the untrained model, at −0.10.

A stronger model still pays for puzzles it has met

GLM also played Prime Agent itself. On Artificial Analysis's Intelligence Index it scores 42 to GPT-6 Luna's 33 at high effort[20]. On ARC at first sight it's the other way round, GLM solving 26.8 percent of the track's public puzzles from three examples and GPT-6 Luna 62.5. Inside the harness the two solve about as often: once every puzzle has come up, GLM solves 95 percent of its visits with its notes wiped, and GPT-6 Luna 93 to 95 percent in its three harnesses.

We read GLM's runs on the visits it solves, since a failed visit costs the whole budget however it failed. Once every puzzle has been met, by round 100, a perfect memory would spend nothing on a solved visit, and whatever a learner spends there is budget its memory could have saved.

From round 100 on, GLM's harness left to decide spent 12.8 percent of the budget on a solved visit, against 17.3 with its notes wiped. A fifth of its solves cost nothing. Given our guidance, the same model in the same harness spent 6.6, and 71 percent of its solves cost nothing. On the seen test it saved a fifth of what it spent with its notes wiped when left to decide, and about half with the guidance.

The same in three harnesses on GPT-6 Luna

We also ran Prime Agent, Hermes Agent and OpenClaw on GPT-6 Luna at three reasoning efforts, in case this was Prime Agent's alone. As shipped, none saved a measurable share on the stream, and at high effort not one of their solves after round 100 was free. Prime Agent never marked a lesson global in 600 sessions, and every session opened on "No saved harness entries yet." OpenClaw, asked to save, wrote 47 to 84 notes a run and read one named for the puzzle on at most 1 percent of the returning visits its log kept. (Even Leonard in Memento read his tattoos.) Told the library's procedure, keeping one Python file per puzzle to run first when it returns, every harness saved a measurable share at every effort, 0.17 to 0.62.

Only the weights got better at new puzzles

On Qwen's held-out test, chance moves none of the panel's learners more than 0.06 from zero, 95 times in 100. No version that keeps notes saved more than that, the most being 0.03 with our guidance, and the program library, which can't carry anything to a puzzle it hasn't met, saved nothing. Learning from the games it won saved 0.09, and self-distillation lost 0.06. Trained to imitate GLM, Qwen saved 0.27 of what the untrained model spent there, solving about 44 of the 100 new puzzles to the untrained model's 14.

Part of it was a page of playing rules

The trained Qwen never answered before seeing an example, which the untrained model did on 40 percent of visits, solving almost none of them. GLM labeled Qwen's moves by a page of six playing rules, among them to look at an example before answering and to submit only a rule, written as a function, that reproduces every example. Told the same page alone, the untrained model answered blind on 6 percent of visits and saved 0.11 on the new puzzles, solving 22 to 26 of them a play. Training gave Qwen more than twice that. Its weights carried more to new puzzles than the rules alone, and on the puzzles it had met they kept far more than the page could, 0.45 against 0.16.

With two and four users at once, each brings 25 puzzles of their own and keeps a harness of their own, while a trained model's users share one adapter. Imitating GLM, the shared model saved 0.30 and 0.34 on new puzzles, and no notes saved more than chance at either.

Why we expected notes to remember and not generalize

A harness's notes are hindsight. A note holds what its writer, the model or its teacher, concluded from a game. Read back, it can reproduce a rule, but it can't make the model better at working out the next one, because the model is the same model. Training on the same games changes the model itself. We expected getting better at new puzzles to come from there. On this stream it was, and the change went beyond the way of playing that a page of rules can give.

A harness could still prove us wrong, most likely one that refactors what it keeps into small functions it can reuse. Re-ARC writes all 400 training rules from one library of 160 primitive functions, and program synthesis has learned libraries like that before[15][19].

What carries over to real harnesses

Office work has no task ids to file programs by, but we think three lessons from ARC still hold for any harness.

• Store liberally, the examples paid for included, and prune later.
• Store functions, not just notes, checked against the examples kept.
• Bring back what's stored without being asked, by what the current work looks like, which OpenClaw didn't.

On this stream, though, only training made the model better at puzzles it hadn't seen. For an agent in deployment, we'd train the model that drives the harness as well as give it a memory.

Limitations and what's next

The stream comes from ARC-AGI-1's public training set, which every model developer knows, and a model may know some of these rules already. Notes and weights are compared on one open student, with one teacher and one seed. None of our models is the strongest available, and a stronger one might keep notes that answer a returning puzzle. Next is Drift, in which a puzzle's rule changes quietly partway through the stream and what a learner keeps can go stale.

The stream's answers and Re-ARC's solvers are public, and a learner could bring them along and score without learning. We verify a result by rerunning it on 50 new puzzles kept private. Submit yours now.

You can open each of GPT-6 Luna’s 228 runs in Continual-ARC’s replay, and every figure and table of the paper is on its Figures page.