Skip to content

Record a folder per run

rb records each run folder from what its config says, and points out a run whose config reports uncommitted code.

Paste this into your agent

Record each run folder in results/ as a run, from what its own config says.

Each folder in results/ is one run of eval.py. Record them with Rabbit Brain.

- Run `rb context --model <your model id>` first, then `rb reconcile`.
- Record the question and the experiment, with each setting cited to the config that states it, and the metric.
- Run `rb reconcile` again, and record each folder with the `rb run add` line it prints, word for word.
- Compute nothing yourself: rb computes the findings.

Finish with `rb reconcile` and `rb status` exactly as printed.

The repository scores a small model on 500 grade-school maths problems with a plain prompt and with three worked examples, two seeds each. The fourth run’s files:

Terminal window
$ cat results/r04/config.yaml
command: python eval.py --config configs/examples.yaml --seed 2 --out results/r04
commit: 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc
dirty: true
started_at: 2026-10-28T11:30:00Z
prompt: examples
problems: 500
seed: 2
Terminal window
$ cat results/r04/metrics.json
{
"accuracy": 46.9,
"finished_at": "2026-10-28T11:53:00Z"
}

rb status in your own terminal, once the agent is done:

Terminal window
$ rb status
Worked examples in the prompt · worked-examples-in-the-prompt
Kept with rb since 2026-11-10. All 4 runs were recorded afterwards, not as they happened.
13 record files not committed
Do three worked examples in the prompt raise accuracy on grade-school maths? (open)
No criterion is linked directly to this question.
Worked examples vs a plain prompt · open
Rules: none yet
The design of 1 experiment is not confirmed: rb confirm lists it.
Facts that ask for nothing (1)
- examples: runs report examples.accuracy, plain.accuracy, which no recorded criterion reads
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.
rb checked the record with git at 2026-11-10 15:32 UTC: all 13 record files are new, changed or deleted since commit 855f30cb64ac, which does not hold the record.
Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/examples/experiment.yaml (new) and 10 more
To check it yourself, in your terminal: git -C ~/prompt-eval status --short --untracked-files=all -- .rb
Experiments
examples Worked examples vs a plain prompt · open · plain (baseline) vs examples (candidate) · varies prompt · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 4
to run again: code partial · environment unknown · files ok
Runs: 4 recorded on 1 experiment · every receipt is what rb wrote
Next: rb confirm
  • “environment unknown”: rb was not watching when these runs were made, so it cannot say which machine made them.
  • A criterion is a rule a result must meet. None is recorded yet: no file here states what counts.
  1. Read the repository

    A folder with a run’s own config beside a results file is one run. rb reconcile prints an rb run add line for each, and writes nothing. Until an experiment is recorded, the line reads <experiment>.

    Terminal window
    $ rb reconcile
    The repository at commit 855f30cb64ac: 4 results folders.
    4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add <experiment> --from <its file> records a run from one.
    Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:
    results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)
    rb run add '<experiment>' --from results/r01/metrics.json --config results/r01/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1
    results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)
    rb run add '<experiment>' --from results/r02/metrics.json --config results/r02/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1
    results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)
    rb run add '<experiment>' --from results/r03/metrics.json --config results/r03/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1
    results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran
    rb run add '<experiment>' --from results/r04/metrics.json --config results/r04/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1
    Settings that differ between them (2):
    prompt: plain (r01, r03), examples (r02, r04)
    seed: 1 (r01, r02), 2 (r03, r04)
  2. Record the question, experiment and settings

    Terminal window
    $ rb question add "Do three worked examples in the prompt raise accuracy on grade-school maths?" --id q-examples
    q-examples: Do three worked examples in the prompt raise accuracy on grade-school maths?
    Next: rb experiment add "<what would answer it>" -q q-examples --hypothesis "<what you expect>"
    Terminal window
    $ rb experiment add "Worked examples vs a plain prompt" --id examples -q q-examples --baseline plain --candidate examples --varies prompt
    Experiment examples: Worked examples vs a plain prompt · plain (baseline) vs examples (candidate) · varies prompt · under q-examples
    Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
    Made 2 variants: plain (baseline), examples (candidate); a setting of one is rb spec set examples plain.<setting> <value>.
    examples declares it varies prompt, but no variant has a value for it yet; give each variant its own: rb spec set examples <variant>.prompt <value> --source <file#key>
    Next: rb spec set examples --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e examples -q q-examples --metric <name> --at-most <x>
    Terminal window
    $ rb spec set examples plain.prompt plain --source "configs/plain.yaml#prompt"
    examples/plain.prompt = plain · verified in configs/plain.yaml#prompt
    $ rb spec set examples examples.prompt examples --source "configs/examples.yaml#prompt"
    examples/examples.prompt = examples · verified in configs/examples.yaml#prompt
    $ rb spec set examples problems 500 --source "configs/plain.yaml#problems"
    examples/problems = 500 · verified in configs/plain.yaml#problems
    $ rb spec set examples seed --per-run
    examples/seed · per-run: each run gives its own, with --set seed=<value> on rb run add
    Terminal window
    $ rb metric add accuracy --unit % --maximize
    Metric accuracy (%): higher is better
  3. Read the folders again

    Terminal window
    $ rb reconcile
    The repository at commit 855f30cb64ac: 4 results folders.
    4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add <experiment> --from <its file> records a run from one.
    Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:
    results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)
    rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1
    results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)
    rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1
    results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)
    rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1
    results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran
    rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1
    Settings that differ between them (2):
    prompt: plain (r01, r03), examples (r02, r04)
    seed: 1 (r01, r02), 2 (r03, r04)
  4. Record each folder

    Terminal window
    $ rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command "python eval.py --config configs/plain.yaml --seed 1 --out results/r01" --command-source results/r01/config.yaml:1
    run3yui3v on examples: plain.accuracy=41.2 · seed=1
    Ran with the spec: 3 of its settings found in results/r01/config.yaml, all matching.
    Produced: 2026-10-28 10:20:00 (reported)
    rb put the code in git at refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1 removes it.
    Since this session last looked:
    - 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/r02/, results/r03/, results/r04/ (rb reconcile)
    Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric plain.accuracy --at-most <x> (write the criterion before the next run)
    $ rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command "python eval.py --config configs/examples.yaml --seed 1 --out results/r02" --command-source results/r02/config.yaml:1
    runqtkwtb on examples: examples.accuracy=47.8 · seed=1
    Ran with the spec: 3 of its settings found in results/r02/config.yaml, all matching.
    Produced: 2026-10-28 10:52:00 (reported)
    Since this session last looked:
    - 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/r03/, results/r04/ (rb reconcile)
    Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric examples.accuracy --at-most <x> (write the criterion before the next run)
    $ rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command "python eval.py --config configs/plain.yaml --seed 2 --out results/r03" --command-source results/r03/config.yaml:1
    runsoyo2w on examples: plain.accuracy=40.6 · seed=2
    Ran with the spec: 3 of its settings found in results/r03/config.yaml, all matching.
    Produced: 2026-10-28 11:21:00 (reported)
    Since this session last looked:
    - results/r04/ holds results written before the record began, which no recorded experiment reads yet (rb reconcile)
    Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric plain.accuracy --at-most <x> (write the criterion before the next run)
    $ rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command "python eval.py --config configs/examples.yaml --seed 2 --out results/r04" --command-source results/r04/config.yaml:1
    runfvajdy on examples: examples.accuracy=46.9 · seed=2
    Ran with the spec: 3 of its settings found in results/r04/config.yaml, all matching.
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran.
    Produced: 2026-10-28 11:53:00 (reported)
    Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric examples.accuracy --at-most <x> (write the criterion before the next run)
  5. Read the repository again

    Terminal window
    $ rb reconcile
    The repository at commit 855f30cb64ac: 4 results folders.
    Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:
    results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)
    recorded as run3yui3v: rb show run3yui3v
    results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)
    recorded as runqtkwtb: rb show runqtkwtb
    results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)
    recorded as runsoyo2w: rb show runsoyo2w
    results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran
    recorded as runfvajdy: rb show runfvajdy
    Settings that differ between them (2):
    prompt: plain (r01, r03), examples (r02, r04)
    seed: 1 (r01, r02), 2 (r03, r04)
    results/r01/ read by the runs of examples
    files: 2 results files
    git: both report and results files are tracked · folder unchanged since the last commit
    results/r02/ read by the runs of examples
    files: 2 results files
    git: both report and results files are tracked · folder unchanged since the last commit
    results/r03/ read by the runs of examples
    files: 2 results files
    git: both report and results files are tracked · folder unchanged since the last commit
    results/r04/ read by the runs of examples
    files: 2 results files
    git: both report and results files are tracked · folder unchanged since the last commit

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0