Record one table of many runs
Your training script adds a row to results.tsv after each run. rb records each row as its own run, and checks the claim in RESULTS.md against those runs.
Paste this into your agent
Section titled “Paste this into your agent”Record every row of results.tsv as a run, and check what RESULTS.md says.
My runs are the rows of results.tsv. Record them with Rabbit Brain, and check what RESULTS.md says against them. - Run `rb context --model <your model id>` first, then `rb reconcile`. - Record the question and the experiment, with each setting cited to the config that states it. - Record every row of results.tsv as its own run, one `rb run add ... --every-row` per arm, with each row's seed and finish time. - Turn the claim in RESULTS.md into the criterion it states, cited to its line. - Compute nothing yourself: rb computes the findings. Finish with `rb reconcile` and `rb status` exactly as printed.
The repository trains a ViT-Tiny on CIFAR-100 with no label smoothing and with smoothing 0.1, four seeds each:
$ head -n 4 results.tsvsmoothing seed val_acc finished_at0.0 1 71.42 2026-10-27T09:00:00Z0.1 1 72.06 2026-10-27T10:00:00Z0.0 2 71.18 2026-10-27T11:00:00Z$ cat RESULTS.md# Results
Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.What you’ll see
Section titled “What you’ll see”rb status in your own terminal, once the agent is done:
$ rb statusLabel smoothing on a small ViT · label-smoothing-on-a-small-vitKept with rb since 2026-11-10. All 8 runs were recorded afterwards, not as they happened.18 record files not committed
Does label smoothing 0.1 raise validation accuracy on ViT-Tiny? (open) Label smoothing 0.1 raises validation accuracy by about 0.66 points. change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005; rule and question link declared after the evidence existed: all 4 units were produced before it was declared. units: 4 seeds from 8 runs; paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit; no run config was compared with the spec (record runs with --config); the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval.
Rules: 1 reproduced, on the estimate alone
The design of 1 experiment is not confirmed: rb confirm lists it.
Facts that ask for nothing (1) - ls: 1 criterion was declared after its evidence existed
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.rb checked the record with git at 2026-11-10 15:32 UTC: all 18 record files are new, changed or deleted since commit d17af2687869, which does not hold the record.Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/ls/criteria/ls-gain-claim.yaml (new) and 15 moreTo check it yourself, in your terminal: git -C ~/label-smoothing status --short --untracked-files=all -- .rb
Rules ls-gain-claim Label smoothing 0.1 raises validation accuracy by about 0.66 points reproduced on the estimate · gain · change.val_acc = 0.66 ± 0.005 reads: change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (higher is better) rule: reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005 timing: declared after the evidence existed: all 4 units were produced before it was declared units: 4 seeds from 8 runs · paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit · no run config was compared with the spec (record runs with --config) · the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval
Experiments ls Label smoothing 0.1 vs none · open · none (baseline) vs ls01 (candidate) · varies smoothing · not frozen · settings 5 verified · 0 provisional · 0 unknown · 1 per-run · runs 8 to run again: code unknown · environment unknown · files ok
Runs: 8 recorded on 1 experiment · every receipt is what rb wrote
Next: rb confirm- A criterion is a rule a result must meet. The claim in
RESULTS.mdis recorded as the criterionls-gain-claim. ls-gain-claimreads reproduced on the estimate: +0.66 is within 0.005 of 0.66, but the interval is wider.- “no run config was compared with the spec”: the runs came from rows, with no
--config.
What rb did, step by step
Section titled “What rb did, step by step”-
Read the repository
Terminal window $ rb reconcileThe repository at commit d17af2687869: 2 results files in the top folder.Results files in no results folder (2):RESULTS.md: prose: rb reads no run from it; a number there is a cited rulea number it states is checked by a cited rule: rb criterion add "<what it claims>" -e <experiment> --metric <metric> --equals <the number> --tolerance <half its last digit> --source RESULTS.md --quote "<its words>"results.tsv: a TSV table of 8 rows, numbers in smoothing, seed, val_acca row is a run: rb run add <experiment> --from results.tsv --where <column>=<value>,... -
Record the question, experiment and settings
Terminal window $ rb question add "Does label smoothing 0.1 raise validation accuracy on ViT-Tiny?" --id q-lsq-ls: Does label smoothing 0.1 raise validation accuracy on ViT-Tiny?Next: rb experiment add "<what would answer it>" -q q-ls --hypothesis "<what you expect>"Terminal window $ rb experiment add "Label smoothing 0.1 vs none" --id ls -q q-ls --baseline none --candidate ls01 --varies smoothingExperiment ls: Label smoothing 0.1 vs none · none (baseline) vs ls01 (candidate) · varies smoothing · under q-lsPlanned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.Made 2 variants: none (baseline), ls01 (candidate); a setting of one is rb spec set ls none.<setting> <value>.ls declares it varies smoothing, but no variant has a value for it yet; give each variant its own: rb spec set ls <variant>.smoothing <value> --source <file#key>Next: rb spec set ls --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e ls -q q-ls --metric <name> --at-most <x>Terminal window $ rb spec set ls --from configs/train.yaml --keys "optim.*,data.*"ls: 3 settings from configs/train.yaml @d17af2687869 · 3 verified$ rb spec set ls none.smoothing 0.0 --source "configs/none.yaml#smoothing"ls/none.smoothing = 0.0 · verified in configs/none.yaml#smoothing$ rb spec set ls ls01.smoothing 0.1 --source "configs/ls01.yaml#smoothing"ls/ls01.smoothing = 0.1 · verified in configs/ls01.yaml#smoothing$ rb spec set ls seed --per-runls/seed · per-run: each run gives its own, with --set seed=<value> on rb run addTerminal window $ rb metric add val_acc --unit % --maximizeMetric val_acc (%): higher is better -
Record every row, one command per arm
--where smoothing=0.0picks one arm’s four rows, and--every-rowrecords each as its own run.--set seed=@seedand--produced-at @finished_atread each run’s seed and time from its row. A row already recorded is skipped.Terminal window $ rb run add ls --from results.tsv --where smoothing=0.0 --variant none --every-row --set seed=@seed --produced-at @finished_atrunvnzbsp on ls: none.val_acc=71.42 · seed=1runx7e5ci on ls: none.val_acc=71.18 · seed=2rune332ux on ls: none.val_acc=71.55 · seed=3runjh7ufm on ls: none.val_acc=71.3 · seed=44 of the 4 rows --where picks in results.tsv recorded as runs.rb put the code in git at refs/rb/snapshots/2f2ced2447df1446e5704536fc328b9bc50e2bb1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/2f2ced2447df1446e5704536fc328b9bc50e2bb1 removes it.$ rb run add ls --from results.tsv --where smoothing=0.1 --variant ls01 --every-row --set seed=@seed --produced-at @finished_atrunm66q7u on ls: ls01.val_acc=72.06 · seed=1runr3l6sg on ls: ls01.val_acc=71.87 · seed=2runjrhrjq on ls: ls01.val_acc=72.21 · seed=3runxoh3jt on ls: ls01.val_acc=71.95 · seed=44 of the 4 rows --where picks in results.tsv recorded as runs.A
.jsonlor.ndjsonfile works the same way, one JSON object a row. So does a SQLite table, with--from results.db --table evals. -
Check the report’s claim
Terminal window $ rb compare lsls Label smoothing 0.1 vs none · varies smoothing · 8 runs under the current spec · means over seedsvariant role smoothing val_acc (%)none baseline 0.0 71.36 (n=4)ls01 candidate 0.1 72.02 +0.66 better (n=4)Terminal window $ rb criterion add "Label smoothing 0.1 raises validation accuracy by about 0.66 points" -e ls -q q-ls --metric change.val_acc --equals 0.66 --tolerance 0.005 --source RESULTS.md --quote "Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds." --id ls-gain-claimls-gain-claim: Label smoothing 0.1 raises validation accuracy by about 0.66 pointschange.val_acc = 0.66 ± 0.005 · primary · on lscited: RESULTS.md "Label smoothing 0.1 raises validation accuracy by about 0..." · states 0.66 (read 'Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.')declared after the evidence existed: all 4 units were produced before it was declaredit already reads change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb): reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005Next: rb run add ls --from <the results file a run wrote> --set seed=<value>Terminal window $ rb show ls-gain-claimCriterion ls-gain-claim on ls (primary, cited): Label smoothing 0.1 raises validation accuracy by about 0.66 pointscited: RESULTS.md "Label smoothing 0.1 raises validation accuracy by about 0..." · states 0.66, read 'Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.'ls-gain-claim Label smoothing 0.1 raises validation accuracy by about 0.66 pointsreproduced on the estimate · gain · change.val_acc = 0.66 ± 0.005reads: change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb)result: gain: the interval clear of 0 on the better side (higher is better)rule: reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005timing: declared after the evidence existed: all 4 units were produced before it was declaredunits: 4 seeds from 8 runsintegrity: no run config was compared, as said below· paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit· no run config was compared with the spec (record runs with --config)· the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole intervalrunvnzbsp: none.val_acc 71.42 (row) · unit seed=1 · produced before the declaration · seed=1runx7e5ci: none.val_acc 71.18 (row) · unit seed=2 · produced before the declaration · seed=2rune332ux: none.val_acc 71.55 (row) · unit seed=3 · produced before the declaration · seed=3runjh7ufm: none.val_acc 71.3 (row) · unit seed=4 · produced before the declaration · seed=4runm66q7u: ls01.val_acc 72.06 (row) · unit seed=1 · produced before the declaration · seed=1runr3l6sg: ls01.val_acc 71.87 (row) · unit seed=2 · produced before the declaration · seed=2runjrhrjq: ls01.val_acc 72.21 (row) · unit seed=3 · produced before the declaration · seed=3runxoh3jt: ls01.val_acc 71.95 (row) · unit seed=4 · produced before the declaration · seed=4 -
Read the repository again
Terminal window $ rb reconcileThe repository at commit d17af2687869: 2 results files in the top folder.Results files in no results folder (2):RESULTS.md: prose: rb reads no run from it; a number there is a cited rulea number it states is checked by a cited rule: rb criterion add "<what it claims>" -e <experiment> --metric <metric> --equals <the number> --tolerance <half its last digit> --source RESULTS.md --quote "<its words>"results.tsv: a TSV table of 8 rows, numbers in smoothing, seed, val_acc; 8 of 8 rows recorded as runsa row is a run: rb run add <experiment> --from results.tsv --where <column>=<value>,...
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0