Continual-ARC · Run yours

Run Continual-ARC on your own learner.

The stream is public and every schedule is deterministic, so there is no held-out set to send a system to. You run it yourself, check the run, and open a pull request, and it joins the leaderboard marked reported until we rerun it from your code.

unifyai/continual-arcThe benchmark: the environment, the schedules, the scoring and the command line
unifyai/continual-arc-baselinesThe paper's learners, the harness adapters, the sandbox and the tracking proxy

Step 01

Install the benchmark

You need uv and Python 3.10 or newer, 3.12 for ARC-AGI-3's games. The last command clones Re-ARC, ARC-AGI and ARC-AGI-2 at pinned commits and downloads the pinned ARC-AGI-3 games, into ~/.cache/continual-arc unless CONTINUAL_ARC_DATA says otherwise.

shell

git clone https://github.com/unifyai/continual-arc
cd continual-arc
uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch

Step 02

A smoke run

arc1-smoke is a short stream, 128 instances of 16 tasks, and the learner here gives up on every instance, so it needs no model and no key and takes about a second. The runner writes the run directory: records.jsonl, one line per instance, with summary.json, schedule.json, config.json and meta.json.

shell

uv run continual-arc schedule make --config arc1-smoke --mode workstreams --seed 0 --out smoke.json
uv run continual-arc run --learner continual_arc.testing:GiveUpLearner \
  --schedule smoke.json --out runs/smoke
uv run continual-arc report runs/smoke

The report's score is 100 minus the budget used, so the leaderboard's U is 100 minus the score, or 100 times the mean cost over 16; its solve rate is P.

Step 03

Write a learner

A learner is any object with these methods, and whatever it keeps between instances is the method under test: notes, programs, a cache, or the weights of a model it trains. The runner calls reset wherever the schedule wipes the learner, which the fresh reference does before every instance. This one keeps the demonstrations it buys for each task and asks solve for an answer; put your method there and it runs as it stands.

my_pkg/learners.py

from continual_arc.env import DemosFeedback, GiveUp, Observation, RequestDemos, Submit
from continual_arc.learner import BaseLearner


def solve(pairs, grid):
    """Your method: an answer from the task's pairs so far, or None to buy one more."""
    return None


class MyLearner(BaseLearner):
    def __init__(self):
        self.pairs = {}                  # what it keeps between instances, by task id

    def reset(self):                     # the fresh reference wipes it before each instance
        self.pairs.clear()

    def act(self, obs: Observation):     # obs.task_id, obs.input, obs.exposure_index, ...
        guess = solve(self.pairs.get(obs.task_id, []), obs.input)
        if guess is not None:
            return Submit(guess)         # wrong costs 1; right ends the instance
        if obs.demo_requests_used < obs.demo_cap:
            return RequestDemos()        # one more pair of the task, costs 1
        return GiveUp()                  # ends the instance, charged as failed

    def observe(self, obs, action, feedback):
        if isinstance(feedback, DemosFeedback):
            self.pairs.setdefault(obs.task_id, []).extend(feedback.pairs)

    def resources(self):                 # cumulative counters, logged and never scored
        return {"pairs_kept": float(sum(map(len, self.pairs.values())))}

Run it from the folder that holds my_pkg/; PYTHONPATH=. lets the runner import it, or install your package into the same environment.

shell

PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule smoke.json --out runs/my-smoke

Step 04

Or drive a harness

The baselines repository runs the paper's learners and drives agent harnesses through the benchmark's protocol. arc-baselines run starts a local proxy for the run and points OPENAI_BASE_URL, OPENROUTER_BASE_URL and ARC_LLM_BASE_URL at it, so every model call a harness makes is counted whatever the harness does, and the real key stays in the proxy. Each harness runs in a sandbox, bubblewrap on Linux and Seatbelt on macOS, that can write only its own state and reach the network only through the proxy, and it sees each task under an alias drawn for the run. The baselines need Python 3.12 or 3.13.

shell

git clone https://github.com/unifyai/continual-arc-baselines
cd continual-arc-baselines
uv sync
uv run continual-arc data fetch --datasets arc1 arc2
uv run arc-baselines run --learner continual_arc.testing:GiveUpLearner \
  --config arc1-smoke --mode workstreams --seed 0

Adapters for Prime Agent, Hermes Agent and OpenClaw are in systems/, and scripts/launch-cell.sh launches any cell the paper reports, with the harness installed at the version of the paper's Appendix D. This is OpenClaw as shipped on GPT-6 Luna at medium effort:

shell

export OPENROUTER_API_KEY=...
bash scripts/launch-cell.sh acp workstreams 0 arc1-small luna6-medium openclaw

For a harness of your own, subclass SystemLearner, which speaks the protocol, handles re-prompts, restarts and wipes, and leaves you five methods.

my_pkg/harness.py

from continual_arc_baselines.systems.base import SystemLearner


class MyHarnessLearner(SystemLearner):
    name = "my-harness"

    def start(self):            # launch it, its model client on the proxy (self.environ)
        ...

    def stop(self):
        ...

    def ask(self, message, *, reprompt=False):  # one message in, its final reply out
        ...

    def new_session(self):      # a fresh conversation; what it keeps stays
        ...

    def wipe_state(self):       # delete everything it kept (called while stopped)
        ...

"end": "finish" ends an instance as the paper's harness rows did: the harness hears the verdict and nothing else, keeps whatever it chooses, and says when it is done. --llm-defaults sets the reasoning effort on any model call that leaves it unset.

shell

PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --config arc1-small --mode workstreams --seed 0 --out runs/arc1/my-harness-ws0

Step 05

Run the canonical stream

Each track has one number: its small configuration in workstreams order at seed 0, with the epilogue of held-out tasks. The leaderboard compares runs at seed 0. The fresh reference serves the same held-out instances with the learner reset before each, and compare reads your forward transfer from the two.

Track	Configuration	Stream, then epilogue
ARC-AGI-1	`arc1-small`	200 instances of 25 training tasks from Re-ARC's generators, then 25 evaluation tasks once each
ARC-AGI-2	`arc2-small`	200 instances of 25 evaluation tasks new to the second edition, each visit in a frame of its own, then 25 more held out
ARC-AGI-3	`arc3-small`	48 levels of six games, then four held-out games, charged in actions against the human baseline

shell

uv run continual-arc schedule make --config arc1-small --mode workstreams --seed 0 \
  --out arc1-ws0.json
uv run continual-arc schedule make --config arc1-small --mode fresh --seed 0 \
  --out arc1-fresh0.json
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-ws0.json --out runs/arc1/my-learner-ws0
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-fresh0.json --out runs/arc1/my-learner-fresh0
uv run continual-arc report runs/arc1/my-learner-ws0
uv run continual-arc compare runs/arc1/my-learner-ws0 runs/arc1/my-learner-fresh0

With its epilogue the ARC-AGI-1 stream is 225 instances. Each of the paper's harness runs on it took 3 to 45 hours and 2 to 63 US dollars of model calls on GPT-6 Luna, as the tracking proxy counted them. For the other tracks, change the configuration.

Step 06

Check it and submit

Fork unifyai/continual-arc, copy the run directory the runner wrote to results/<track>/<name>/, and add a submission.json beside its five files. Add the fresh run the same way, under a name of its own, if you want your transfer read.

submission.json

{
  "name": "my-harness, as shipped",
  "learner": "my-harness 1.4.0, with its memory and skills switched on",
  "model": "GPT-6 Luna, medium effort",
  "code": "https://github.com/you/my-harness",
  "contact": "you@example.com",
  "notes": "anything a reader needs to know to read the number"
}

scripts/check_result.py rebuilds the schedule from the run's configuration, order and seed, checks that the records cover every slot in order and that the summary is the one the records give, and prints the numbers the leaderboard will show. It calls no model and no network.

shell

mkdir -p results/arc1
cp -r runs/arc1/my-learner-ws0 results/arc1/my-learner-ws0
# write results/arc1/my-learner-ws0/submission.json, then
uv run python scripts/check_result.py results/arc1/my-learner-ws0

Then open a pull request. Your result goes on the leaderboard marked reported, and once we have rerun it from your code it is marked verified. Questions go to the repository's issues.