Record a folder per run
rb records each run folder from what its config says, and points out a run whose config reports uncommitted code.
Paste this into your agent
Section titled “Paste this into your agent”Record each run folder in results/ as a run, from what its own config says.
Each folder in results/ is one run of eval.py. Record them with Rabbit Brain. - Run `rb context --model <your model id>` first, then `rb reconcile`. - Record the question and the experiment, with each setting cited to the config that states it, and the metric. - Run `rb reconcile` again, and record each folder with the `rb run add` line it prints, word for word. - Compute nothing yourself: rb computes the findings. Finish with `rb reconcile` and `rb status` exactly as printed.
The repository scores a small model on 500 grade-school maths problems with a plain prompt and with three worked examples, two seeds each. The fourth run’s files:
$ cat results/r04/config.yamlcommand: python eval.py --config configs/examples.yaml --seed 2 --out results/r04commit: 48917c6e0ef4fc881eff8e5fc999ce9aabe62abcdirty: truestarted_at: 2026-10-28T11:30:00Zprompt: examplesproblems: 500seed: 2$ cat results/r04/metrics.json{ "accuracy": 46.9, "finished_at": "2026-10-28T11:53:00Z"}What you’ll see
Section titled “What you’ll see”rb status in your own terminal, once the agent is done:
$ rb statusWorked examples in the prompt · worked-examples-in-the-promptKept with rb since 2026-11-10. All 4 runs were recorded afterwards, not as they happened.13 record files not committed
Do three worked examples in the prompt raise accuracy on grade-school maths? (open) No criterion is linked directly to this question. Worked examples vs a plain prompt · open
Rules: none yet
The design of 1 experiment is not confirmed: rb confirm lists it.
Facts that ask for nothing (1) - examples: runs report examples.accuracy, plain.accuracy, which no recorded criterion reads
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.rb checked the record with git at 2026-11-10 15:32 UTC: all 13 record files are new, changed or deleted since commit 855f30cb64ac, which does not hold the record.Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/examples/experiment.yaml (new) and 10 moreTo check it yourself, in your terminal: git -C ~/prompt-eval status --short --untracked-files=all -- .rb
Experiments examples Worked examples vs a plain prompt · open · plain (baseline) vs examples (candidate) · varies prompt · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 4 to run again: code partial · environment unknown · files ok
Runs: 4 recorded on 1 experiment · every receipt is what rb wrote
Next: rb confirm- “environment unknown”: rb was not watching when these runs were made, so it cannot say which machine made them.
- A criterion is a rule a result must meet. None is recorded yet: no file here states what counts.
What rb did, step by step
Section titled “What rb did, step by step”-
Read the repository
A folder with a run’s own config beside a results file is one run.
rb reconcileprints anrb run addline for each, and writes nothing. Until an experiment is recorded, the line reads<experiment>.Terminal window $ rb reconcileThe repository at commit 855f30cb64ac: 4 results folders.4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add <experiment> --from <its file> records a run from one.Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)rb run add '<experiment>' --from results/r01/metrics.json --config results/r01/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)rb run add '<experiment>' --from results/r02/metrics.json --config results/r02/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)rb run add '<experiment>' --from results/r03/metrics.json --config results/r03/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ranrb run add '<experiment>' --from results/r04/metrics.json --config results/r04/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1Settings that differ between them (2):prompt: plain (r01, r03), examples (r02, r04)seed: 1 (r01, r02), 2 (r03, r04) -
Record the question, experiment and settings
Terminal window $ rb question add "Do three worked examples in the prompt raise accuracy on grade-school maths?" --id q-examplesq-examples: Do three worked examples in the prompt raise accuracy on grade-school maths?Next: rb experiment add "<what would answer it>" -q q-examples --hypothesis "<what you expect>"Terminal window $ rb experiment add "Worked examples vs a plain prompt" --id examples -q q-examples --baseline plain --candidate examples --varies promptExperiment examples: Worked examples vs a plain prompt · plain (baseline) vs examples (candidate) · varies prompt · under q-examplesPlanned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.Made 2 variants: plain (baseline), examples (candidate); a setting of one is rb spec set examples plain.<setting> <value>.examples declares it varies prompt, but no variant has a value for it yet; give each variant its own: rb spec set examples <variant>.prompt <value> --source <file#key>Next: rb spec set examples --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e examples -q q-examples --metric <name> --at-most <x>Terminal window $ rb spec set examples plain.prompt plain --source "configs/plain.yaml#prompt"examples/plain.prompt = plain · verified in configs/plain.yaml#prompt$ rb spec set examples examples.prompt examples --source "configs/examples.yaml#prompt"examples/examples.prompt = examples · verified in configs/examples.yaml#prompt$ rb spec set examples problems 500 --source "configs/plain.yaml#problems"examples/problems = 500 · verified in configs/plain.yaml#problems$ rb spec set examples seed --per-runexamples/seed · per-run: each run gives its own, with --set seed=<value> on rb run addTerminal window $ rb metric add accuracy --unit % --maximizeMetric accuracy (%): higher is better -
Read the folders again
Terminal window $ rb reconcileThe repository at commit 855f30cb64ac: 4 results folders.4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add <experiment> --from <its file> records a run from one.Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ranrb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1Settings that differ between them (2):prompt: plain (r01, r03), examples (r02, r04)seed: 1 (r01, r02), 2 (r03, r04) -
Record each folder
Terminal window $ rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command "python eval.py --config configs/plain.yaml --seed 1 --out results/r01" --command-source results/r01/config.yaml:1run3yui3v on examples: plain.accuracy=41.2 · seed=1Ran with the spec: 3 of its settings found in results/r01/config.yaml, all matching.Produced: 2026-10-28 10:20:00 (reported)rb put the code in git at refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1 removes it.Since this session last looked:- 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/r02/, results/r03/, results/r04/ (rb reconcile)Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric plain.accuracy --at-most <x> (write the criterion before the next run)$ rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command "python eval.py --config configs/examples.yaml --seed 1 --out results/r02" --command-source results/r02/config.yaml:1runqtkwtb on examples: examples.accuracy=47.8 · seed=1Ran with the spec: 3 of its settings found in results/r02/config.yaml, all matching.Produced: 2026-10-28 10:52:00 (reported)Since this session last looked:- 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/r03/, results/r04/ (rb reconcile)Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric examples.accuracy --at-most <x> (write the criterion before the next run)$ rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command "python eval.py --config configs/plain.yaml --seed 2 --out results/r03" --command-source results/r03/config.yaml:1runsoyo2w on examples: plain.accuracy=40.6 · seed=2Ran with the spec: 3 of its settings found in results/r03/config.yaml, all matching.Produced: 2026-10-28 11:21:00 (reported)Since this session last looked:- results/r04/ holds results written before the record began, which no recorded experiment reads yet (rb reconcile)Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric plain.accuracy --at-most <x> (write the criterion before the next run)$ rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command "python eval.py --config configs/examples.yaml --seed 2 --out results/r04" --command-source results/r04/config.yaml:1runfvajdy on examples: examples.accuracy=46.9 · seed=2Ran with the spec: 3 of its settings found in results/r04/config.yaml, all matching.results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran.Produced: 2026-10-28 11:53:00 (reported)Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric examples.accuracy --at-most <x> (write the criterion before the next run) -
Read the repository again
Terminal window $ rb reconcileThe repository at commit 855f30cb64ac: 4 results folders.Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)recorded as run3yui3v: rb show run3yui3vresults/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)recorded as runqtkwtb: rb show runqtkwtbresults/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)recorded as runsoyo2w: rb show runsoyo2wresults/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ranrecorded as runfvajdy: rb show runfvajdySettings that differ between them (2):prompt: plain (r01, r03), examples (r02, r04)seed: 1 (r01, r02), 2 (r03, r04)results/r01/ read by the runs of examplesfiles: 2 results filesgit: both report and results files are tracked · folder unchanged since the last commitresults/r02/ read by the runs of examplesfiles: 2 results filesgit: both report and results files are tracked · folder unchanged since the last commitresults/r03/ read by the runs of examplesfiles: 2 results filesgit: both report and results files are tracked · folder unchanged since the last commitresults/r04/ read by the runs of examplesfiles: 2 results filesgit: both report and results files are tracked · folder unchanged since the last commit
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0