Compare against a published baseline
A match to a paper counts only on the paper’s settings. Cite its number as a criterion, a rule a result must meet, and cite its settings.
Paste this into your agent
Section titled “Paste this into your agent”Check our run against the paper's number, with the paper's settings cited.
Check our ResNet-20 against the paper's 8.75% error with Rabbit Brain. The paper's numbers are saved in docs/baselines/resnet-paper.md. Count it as reproduced if we land within 0.3 points of the paper's 8.75% error. - Run `rb context --model <your model id>` first. - Record the experiment, and each training setting from configs/train.yaml with the value the paper used, cited to its line in docs/baselines/resnet-paper.md (`--cited`, `--cited-source`). - Record the paper's number as a criterion cited to its table row (`--source`, `--quote`, `--locator "Table 6"`), with my tolerance and my words. - Run seeds 1 to 3 through `rb journal run`, and record each run. Finish with `rb show` of the criterion exactly as printed, and every setting that differs from the paper's.
The paper’s text is saved in docs/baselines/resnet-paper.md:
# Deep Residual Learning for Image Recognition (2016): excerpts
Section 4.2, CIFAR-10:
We use a weight decay of 0.0001 and momentum of 0.9.These models are trained with a mini-batch size of 128.We start with a learning rate of 0.1.
Table 6, classification error on the CIFAR-10 test set:
| method | layers | params | error (%) ||---|---|---|---|| ResNet | 20 | 0.27M | 8.75 || ResNet | 32 | 0.46M | 7.51 |What you’ll see
Section titled “What you’ll see”$ rb show matches-paper-errorCriterion matches-paper-error on repro (primary, cited): Our ResNet-20 lands within 0.3 points of the paper's 8.75% error · words (as given): "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error." cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75, read '| ResNet | 20 | 0.27M | 8.75 |' matches-paper-error Our ResNet-20 lands within 0.3 points of the paper's 8.75% error reproduced · measured · resnet20.error = 8.75 ± 0.3 (n ≥ 3) reads: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb) result: measured: resnet20 on its own, with its interval, so neither better nor worse rule: reproduced: estimate 8.747 within 8.75 ± 0.3 timing: declared before the evidence (by production times) units: 3 seeds integrity: ok rune5stgq: resnet20.error 8.81 (file) · unit seed=1 · seed=1 runx5twzo: resnet20.error 8.69 (file) · unit seed=2 · seed=2 runrvxoet: resnet20.error 8.74 (file) · unit seed=3 · seed=3What rb did, step by step
Section titled “What rb did, step by step”-
Record the experiment’s purpose
Terminal window $ rb experiment add "ResNet-20 against the ResNet paper (2016)" --id repro -q q-repro --purpose "Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6" --baseline resnet20Experiment repro: ResNet-20 against the ResNet paper (2016) · resnet20 (baseline) · under q-repropurpose: Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.Made 1 variant: resnet20 (baseline); a setting of one is rb spec set repro resnet20.<setting> <value>.Next: rb spec set repro --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e repro -q q-repro --metric <name> --at-most <x> -
Cite each setting and the paper’s value
--citedkeeps the paper’s value beside yours. A difference makes every criterion that cites the paper not comparable.Terminal window $ rb spec set repro optim.lr --source "configs/train.yaml#optim.lr" --cited 0.1 --cited-source docs/baselines/resnet-paper.md:7repro/optim.lr = 0.1 · verified in configs/train.yaml#optim.lrcited: 0.1 (docs/baselines/resnet-paper.md:7)$ rb spec set repro optim.momentum --source "configs/train.yaml#optim.momentum" --cited 0.9 --cited-source docs/baselines/resnet-paper.md:5repro/optim.momentum = 0.9 · verified in configs/train.yaml#optim.momentumcited: 0.9 (docs/baselines/resnet-paper.md:5)$ rb spec set repro optim.weight_decay --source "configs/train.yaml#optim.weight_decay" --cited 0.0001 --cited-source docs/baselines/resnet-paper.md:5repro/optim.weight_decay = 0.0001 · verified in configs/train.yaml#optim.weight_decaycited: 0.0001 (docs/baselines/resnet-paper.md:5)$ rb spec set repro data.batch_size --source "configs/train.yaml#data.batch_size" --cited 128 --cited-source docs/baselines/resnet-paper.md:6repro/data.batch_size = 128 · verified in configs/train.yaml#data.batch_sizecited: 128 (docs/baselines/resnet-paper.md:6)$ rb spec set repro seed --per-runrepro/seed · per-run: each run gives its own, with --set seed=<value> on rb run addTerminal window $ rb show repro/data.batch_sizerepro/data.batch_size = 128 · verifiedsource: configs/train.yaml#data.batch_size · read 'data.batch_size: 128' @9d87bd151701cited source used 128 (docs/baselines/resnet-paper.md:6)history:2026-11-10T15:32:17 spec set: setting data.batch_size: new, 1282026-11-10T15:32:17 spec check: verified -
Cite the paper’s number as a criterion
rb checks that the quoted row is in the file and states 8.75.
Terminal window $ rb criterion add "Our ResNet-20 lands within 0.3 points of the paper's 8.75% error" -e repro -q q-repro --metric resnet20.error --equals 8.75 --tolerance 0.3 --source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6" --min-n 3 --words "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error." --id matches-paper-errormatches-paper-error: Our ResNet-20 lands within 0.3 points of the paper's 8.75% errorresnet20.error = 8.75 ± 0.3 (n ≥ 3) · primary · on reprocited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75 (read '| ResNet | 20 | 0.27M | 8.75 |')declared; no evidence recorded yetNext: rb run add repro --from <the results file a run wrote> --set seed=<value> -
Run and record each seed
Terminal window $ rb journal run -- python train.py --seed 1{"metrics": {"resnet20.error": 8.81}}exit 0. The command, its exit code and what it wrote are in the journal.It printed its metrics: resnet20.error=8.81. rb reconcile proposes the run of repro that records them.Since this session last looked:- out/seed1/ holds results no recorded experiment reads yet (rb reconcile)- out/seed1/metrics.json: results landed and are not in the record (rb reconcile)Next: rb reconcileTerminal window $ rb run add repro --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"rune5stgq on repro: resnet20.error=8.81 · seed=1Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching.Produced: 2026-11-10 15:32:17 (file)matches-paper-error criterion undetermined · point only: resnet20.error: 8.81 % · mean, 1 seed (rb)rb put the code in git at refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30 removes it.Next: rb show matches-paper-error$ rb run add repro --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"runx5twzo on repro: resnet20.error=8.69 · seed=2Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching.Produced: 2026-11-10 15:32:17 (file)matches-paper-error criterion undetermined · point only: resnet20.error: 8.75 % · mean, 2 seeds (rb)Next: rb show matches-paper-error$ rb run add repro --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"runrvxoet on repro: resnet20.error=8.74 · seed=3Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching.Produced: 2026-11-10 15:32:18 (file)matches-paper-error reproduced · measured: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb)Next: rb show matches-paper-error
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0