The figures and tables of the Continual-ARC paper, in its order, drawn from the records of its 257 runs. Point at a chart to read each line and its interval, switch effort, order and track where the paper has panels, turn learners off and on in the legend, and open the run behind any number in the replay.
A learner meets a long stream of ARC instances. The same tasks come back after dormant gaps, each time with a fresh input, while new tasks keep arriving, and every instance is charged for the feedback it uses.
Figure 1
One instance
The learner sees the task id and a fresh input grid. It may submit a grid, and hears only correct or incorrect, or pay for one more demonstration pair of the task. Each wrong answer and each demonstration costs one, against a cap of sixteen, and the budget used is that cost as a share of the cap, so lower is better. What the learner carries between instances is recorded but does not enter the score.
The exposure axis, and the three orders of the stream
Order
Below, the stream itself: every instance the ARC-AGI-1 track serves, in each order, one row per task. Point at a column to see which task it served.
first meeting with the taskthe task comes backtask active, in a spellheld-out task, served once
Point at a column to see which task it served; the task’s marks light up in every order.
Left: what the exposure curve should look like, sketched without data. A learner that keeps nothing draws a flat line, a cache of demonstrations drops once, by the demonstrations it no longer buys, and a learner that has learned the task falls towards zero. Right: the 200 instances of the ARC-AGI-1 stream in each order, seed 0, one row per task and one mark per instance, with a thin line for each spell in which a task is active and the 25 held-out tasks in grey at the end. In the workstreams order, the track’s own, a learner has met 14 tasks by step 50 and all 25 by step 100, and a task comes back after a median gap of 5 instances, at most 141.
Results on ARC-AGI-1
GPT-6 Luna at low, medium and high reasoning effort drives every learner: the two floors, the program library, and each harness as shipped and with everything it wrote wiped before every instance. Every learner meets the same instances in the same order.
ARC-AGI-1, arc1-small: the share of the feedback budget each learner used, in percent, lower is better. Workstreams is the track’s stream, and the epilogue after it, the 25 held-out tasks, is read against fresh, the same instances with the learner reset before each; there a harness as shipped and wiped run the same protocol, so their spread shows the noise. Sequential and random are the same stream in the two other orders. Small type is the standard error over tasks, the best in each row is bold, even within error bars, and every number opens the replay of its run.
ARC-AGI-1, workstreams order: budget used by the number of times the task had been met before, one panel per effort, lower is better, each panel on its own scale. A learner that relearns every task draws a flat line, and the demo-cache floor falls only by the demonstrations it no longer buys; the dotted line is its mean after the first meeting. A learner that has learned a task falls below it. The program library does, using 12.7 to 29.6 points less of the budget than every harness at every effort; no harness beats the demo-cache floor at any effort (Table 18).
ARC-AGI-1, workstreams order: budget saved against the stateless floor on the same instance, by the number of times the task had been met before, one panel per effort, higher is better and zero is the floor. Pairing on the instance takes out what the schedule itself does, such as how hard the drawn instances are. A line that rises as the task returns shows memory, and the gap between a harness’s solid and dashed lines is what its memory is worth. Pooled over the three orders it runs from −6.1 to 1.0 points (Table 18).
Benchmark details
Appendix A of the paper: the words, the orders, the models, the configurations and the tasks.
Table 2
The terms
term
meaning
task
an ARC rule, known to the learner by its id; on ARC-AGI-3, a game
instance
one input of a task to solve: generated fresh on ARC-AGI-1, freshly augmented on ARC-AGI-2, a level on ARC-AGI-3
slot
one entry of a schedule, naming a task, an instance seed and a phase
phase
the stream, or the epilogue that follows it
epilogue
held-out tasks never seen before, served once each after the stream and unannounced
schedule
the ordered list of slots a learner is served, fixed by a configuration, an order and a seed
configuration
the track, the numbers of tasks and users, the stream and epilogue lengths and the workstreams parameters, such as arc1-small (Table 5)
order
the order in which the same tasks are served: sequential, random or workstreams (the paper calls it the mode)
regime
a setting of the number of users: individual (one user, whose learner may change between any two instances) or collective (a hundred, served in rounds, with changes only between rounds)
track
one fixed schedule per ARC-AGI edition: the small configuration in the workstreams order with its epilogue, one user, seed 0
fresh reference
the epilogue served alone, with the learner reset before each instance, against which forward transfer is measured
exposure index
the number of times the instance's task had been met before
gap
the number of instances since the task was last met
row
a kind of learner: the floors (F), the program library (PL), and the harnesses as shipped (H) and wiped (H0)
preset
a model at one reasoning effort
cell
one run: a row at one preset in one order
The words the paper and these pages use for the parts of the benchmark.
Table 3
The three orders as settings of one process
order
tasks active
spell
rest
new tasks
sequential
1
all its instances
never returns
the next at once
workstreams
a few: four on arc1-small
drawn afresh each time, three instances on average
drawn afresh each time, a median of thirty instances
at random, one every two instances on average
random
all
never ends
none
all at the start
Every stream comes from one process: a few tasks are active at once, each for a spell, then rests before it may return, while new tasks arrive. The three orders are settings of it. Each is seeded by the configuration and the seed, so a stream is the same every time it is built, and the instance served at step t is the same in every order.
Table 4
The models
track
model, effort
verified
USD per task
what ran, in which orders
ARC-AGI-1
GPT-6 Luna, low
37.7%
0.003
every learner: workstreams, sequential, random, fresh
ARC-AGI-1
GPT-6 Luna, medium
61.0%
0.006
every learner: workstreams, sequential, random, fresh
ARC-AGI-1
GPT-6 Luna, high †
70.3%
0.009
every learner: workstreams, sequential, random, fresh
ARC-AGI-2
GPT-6 Luna, low †
4.6%
0.005
floors: random, fresh
ARC-AGI-2
GPT-6 Luna, medium
18.1%
0.012
floors: workstreams, random, fresh; program library: workstreams, fresh; Prime Agent: workstreams, sequential, fresh
ARC-AGI-2
GPT-6 Luna, high
31.4%
0.023
floors: workstreams, random, fresh; program library: workstreams, fresh; Prime Agent: workstreams, sequential, fresh
ARC-AGI-2
GPT-6 Luna, xhigh
41.9%
0.032
floors: workstreams, random, fresh; program library: workstreams, fresh; Prime Agent: workstreams, sequential, fresh
ARC-AGI-2
GPT-6 Luna, max
59.3%
0.062
floors: workstreams, random, fresh; program library: workstreams, fresh; Prime Agent: workstreams, sequential, fresh
ARC-AGI-3
Claude Opus 5.5, low †
–
–
Prime Agent: workstreams, fresh
The presets, chosen from the ARC Prize leaderboard: by default a track runs the efforts of one model whose verified zero-shot score lies between about 15 and 60 percent, the band in which a learner has room to learn. Marked † are the three outside it or not on the board: GPT-6 Luna at high effort on ARC-AGI-1, at low effort on ARC-AGI-2, which runs the floors alone as the low end of the check, and Claude Opus 5.5 on ARC-AGI-3, which the board does not list. Verified is the leaderboard’s semi-private score at that effort and the cost per task the leaderboard’s. What ran is read from the run records; every harness ran as shipped and wiped.
Figure 5
An input grid does not say which task it belongs to
inputoutput under Aoutput under B
A feca6190B 7fe24cdd
A 91413438B 007bbfb7
A c3e719e8B ac0a08a4
Each row is one input generated for training task A, with the output of A’s rule and of task B’s rule on the same grid. Both are valid ARC tasks, and only the demonstrations or the id say which rule applies, which is why every instance shows its task id. Top: A draws diagonal lines from each coloured cell on a larger black square, and B lays out the input and its three rotations as four quadrants. Middle: A tiles copies of the input, one per coloured cell, and B places the input’s pattern in every tile whose cell is coloured. Bottom: A places the whole input in every tile whose cell has the most frequent colour, and B enlarges every cell into a block. The pairs were found automatically: an input generated for A on which B’s verifier runs and returns a different valid grid.
Table 5
The named configurations
configuration
tasks
stream
epilogue
users
active
spell
rest
new task every
arc1-small
25
200
25
1
4
3
30
2
arc1-medium
100
3,000
100
1
5
6
120
20
arc1-full
400
12,000
400
1
6
6
300
20
arc1-collective
25
200
25
100
4
3
30
2
arc2-small
25
200
25
1
4
3
30
2
arc3-small
6
48
4
1
2
3
8
6
Each track is its small configuration in the workstreams order, one user, seed 0; only the small ones have runs. Users is the number served at once, and the stream and epilogue lengths are per user. The last four columns set the workstreams order: tasks active at once, the mean spell, the median rest and the mean time between new tasks, all in instances. Every configuration caps an instance at 8 wrong answers and 8 demonstrations; on ARC-AGI-3 the cap is five times a person’s actions on the level.
The ARC-AGI-1 track: the 25 stream tasks of arc1-small, drawn from the 400 training tasks, and its 25 held-out tasks from the evaluation set. Each thumbnail is the task's first demonstration pair, input beside output. Figures 6, 7 and 8 are one per track; every task opens on the Tasks page with all its pairs and how each learner did on each visit.
Running the harnesses
Appendix D: each harness runs as a black box in a sandbox, with a fresh conversation for every instance and nothing carried over but what it wrote itself.
Table 7
How the adapter drives a harness
rule
Prime Agent
Hermes Agent, OpenClaw
action budget
300 s or 15 model calls a turn
600 s a turn
after a turn cut for budget
300 s or 6 calls, with a notice
600 s, with a notice
longest one turn may run
1,800 s
the action budget
silence before a restart
600 s with no event and no model traffic
300 s with no update and no model traffic
model errors
the harness's own retries, then five resends with backoff from 60 s
the harness's own retries
a reply with no readable action
two re-prompts, then a give-up
the same
empty replies
unreadable; a restart after four in a row
the same
restarts
two an action, files kept, then a give-up
the same
end of an instance
the verdict, then the harness's finish
the same; OpenClaw then idles 40 s
context window declared, reply cap
262,144 and 65,536 tokens
the same
The adapter's rules. A turn cut for budget does not fail the instance: the harness keeps what it said and its next message tells it so. A give-up is scored as a failed instance, charged the attempts it had left, and no learner may choose one. A restart opens a new session, which under one conversation per instance costs the harness nothing it would have kept.
What the harnesses kept
Appendix E reads everything each harness stored in its nine runs as shipped. Left to decide, none stored what would answer a returning task. The post takes the harnesses one at a time.
Table 8
What each learner is built to keep
learner
what it keeps
written by
how it comes back
program library
one Python program per task, kept once it reproduces every pair the model has, and the pairs bought, up to eight a task
the learner, whenever a program passes its checks
run first; the model is asked only when it fails
Prime Agent
memories, prompt notes, functions, skills and sub-agents
the agent, when it chooses; an automatic refinement every 25 turns
a digest of the store, capped in size, at the start of a session; a search tool
Hermes Agent
skill files and a memory about its user
a background review after a turn with ten or more tool calls; the agent, when it chooses
the agent reads its skills at the start of a session or when it wants them
OpenClaw
a consolidated memory, skills and workspace files
a scheduled consolidation over its transcripts and a skill workshop; the agent, when it chooses
skills listed at the start of a session; search over its memory and over the conversation of every earlier instance
What each learner is built to keep between instances, who writes it, and how it reaches the model when a task returns, read from the designs rather than from any run. The three harnesses are built for lessons that carry across unrelated work, and none of them files what it keeps under the task id, as the program library does.
Table 9
What memory changes in how a harness spends
pairs bought
wrong answers
solved (%)
budget used (points)
harness
effort
first
returns
returns
returns
first
returns
Prime Agent
low
−0.28
−0.02
+0.02
+0.4
−3.0 ±2.8
0.0 ±1.2
medium
+0.19
−0.03
−0.02
+1.3
+1.8 ±2.4
−0.3 ±0.7
high
+0.03
+0.03
−0.03
0.0
+0.9 ±1.7
0.0 ±0.8
Hermes Agent
low
+0.87
+0.64
+0.30
−6.1
+7.8 ±3.6
+5.9 ±1.4
medium
+0.16
+0.32
−0.19
+0.8
+1.1 ±2.8
+0.8 ±1.1
high
+0.04
−0.05
−0.13
+1.3
+0.3 ±2.9
−1.1 ±1.0
OpenClaw
low
+0.61
+0.38
−0.13
+1.9
+5.7 ±3.1
+1.5 ±1.3
medium
+0.24
+0.24
+0.07
−1.1
+2.9 ±1.5
+1.9 ±0.8
high
+0.17
+0.02
+0.01
+0.2
+0.7 ±2.7
+0.2 ±0.7
ARC-AGI-1: each harness as shipped minus itself wiped, on the same instances, pooled over the three orders, at first meetings (75 instances) and at returns (525), per instance. A positive budget is a cost of memory, and ± is one standard error over tasks. Pairs and wrong answers are counts, and one of either is 6.25 points of budget. A change at a task’s first meeting, such as the extra demonstrations Hermes Agent and OpenClaw buy at low effort, cannot come from memory of the task: it comes from what the harness kept about how to work.
Table 10
What each harness stored, and what it saved
harness
what it held after the stream
what reached the model on a return
saved, low
saved, medium
Prime Agent
nothing beyond the session: no global store was ever created
a digest of the empty store, the same in every session
What each harness held after the medium-effort workstreams stream, what of it reached the model when a task returned, and what its memory saved against itself wiped, in points of the budget on the workstreams stream alone, ± one standard error over tasks. Table 18 gives the savings pooled over the three orders, with 95 percent intervals.
Asked to store, and told to keep a program library
Appendix F changes exactly what the main rows leave alone: one message after every verdict asks the harness to store anything that could help, or its rules tell it to keep a program per task as the program library does.
Table 11
The ablation design
who acts
how it answers
nothing kept
kept between instances
bare model
as it chooses
stateless floor (F)
demo-cache floor (F): the pairs shown again
bare model
a program for every answer
program library, nothing kept (PL0)
program library (PL): the pairs, and the kept program run first
harness
as it chooses
wiped (H0)
as shipped (H): whatever the harness keeps
harness
as it chooses, asked to store
not run
asked to store (HA)
harness
the program library, as a procedure
told to keep a program library, wiped (HPL0)
told to keep a program library (HPL): the programs it saves
The rows of the ablations on ARC-AGI-1, as a design of three factors: who acts, how it answers, and whether anything is kept between instances. The floors, the program library and the harnesses as shipped and wiped are the main rows; the others appear only in the ablations.
ARC-AGI-1: the budget each row saves against its reference, in points, on the same stream instances, pooled over the three orders, with a standard error over tasks (the whisker, plus or minus one). A harness row is read against the same harness wiped before every instance, except the last two lines of each harness, which read the harness told to keep a program library against itself wiped and against the program library; the bare model’s rows are read as named. The bars share one scale, from −25 to +35 points.
ARC-AGI-1: budget used by the number of times the task had been met before, for each harness’s ablation rows, pooled over the three orders, one row of panels per harness and one column per effort, each panel on its own scale. The harness wiped and the program library are the anchors, and a row that learns from what it keeps falls from the first towards the second as a task returns. Told to keep a program library, Prime Agent saves 12.0 to 15.0 points against itself wiped, and asked to store, 3.6 to 6.7; OpenClaw, asked to store, saves nothing at any effort (from −2.3 to 0.4). A point is drawn once at least ten instances fall in it.
Figure 10
The bare model with and without programs, by exposure
ARC-AGI-1: the bare model, pooled over the three orders, one panel per effort: the two floors, answering as it chooses with nothing kept and with its demonstrations kept, and the program library with nothing kept and with its programs and demonstrations kept. A program for every answer saves 5.5 to 17.6 points against the stateless floor with nothing kept, and keeping the programs and demonstrations saves 13.3 to 13.7 more, most of it as a task returns.
All results
Appendix H: every table and figure for each track, with an epilogue after every order.
ARC-AGI-1: budget used, in percent, lower is better, as in Table 1 but with the epilogue after every order. The same 25 held-out tasks end each order, and fresh is the reference for all three epilogues.
Table 18
What each harness saves, with 95 percent intervals
ARC-AGI-1: the budget each harness saves, in points, paired instance by instance with the learner it is set against and pooled over the three orders (600 instances), with a 95 percent interval by bootstrap over tasks. Negative means the harness used more. The bars share one scale, from −15 to +10 points. No harness beats the demo-cache floor at any effort, and none gains measurably from its memory; the only intervals that exclude zero are costs.
ARC-AGI-1, workstreams order: the share of instances solved within the cap by the number of times the task had been met before, one panel per effort, all on one scale; the solve-rate twin of Figure 3. The leftmost point is the first-meeting solve rate, the check that a preset has room to learn. The two views differ where a learner learns to solve but keeps paying for attempts and demonstrations on the way, and where it fails outright.
Figure 12
Budget used by exposure, in the other two orders
Order
Effort
Sequential
Random
demo-cache floor after its first meeting
Sequential · GPT-6 Luna, low · exposure 7met 7 times before
ARC-AGI-1: budget used by the number of times the task had been met before, in the sequential order, each task’s instances served together, and the random order, drawn evenly over the stream, one column per effort, as Figure 3 is for the workstreams order. Every number of both orders is in Tables 14 to 17.
ARC-AGI-2, arc2-small: budget used, in percent, lower is better, in the layout of Table 14. Instances are the public pairs of evaluation tasks new to the second edition, under the transformations each task’s rule survives. Prime Agent is the only harness on this track, and rows that no learner ran are left out: at low effort only the floors ran, in the random order and fresh.
ARC-AGI-2, workstreams order: budget used by the number of times the task had been met before, one panel per effort of GPT-6 Luna, as Figure 3. Paired on the same instances and pooled over the workstreams and sequential orders, Prime Agent’s memory saves from −0.2 to 1.5 points against itself wiped, and every 95 percent interval includes zero.
ARC-AGI-3, arc3-small, Claude Opus 5.5 at low effort: budget used, in percent of the action cap, lower is better. The stream serves successive levels of each game, and the epilogue and fresh first levels only, so the stream row is not comparable with the two under it. Three games came round again, and on the 15 levels met a second time Prime Agent as shipped used 31 percent of the budget against 24 at the first meeting, and wiped 32 against 40: no sign of a level played better the second time, though with one run of each row the reading is tentative.
Not drawn here: the paper’s Algorithm 1, the stream process written out, and its Tables 6 and 13, the software versions behind each row and the comparison with related benchmarks. They are in the paper.