Skip to content

Compare against a published baseline

A match to a paper counts only on the paper’s settings. Cite its number as a criterion, a rule a result must meet, and cite its settings.

Paste this into your agent

Check our run against the paper's number, with the paper's settings cited.

Check our ResNet-20 against the paper's 8.75% error with Rabbit Brain. The paper's numbers are saved in docs/baselines/resnet-paper.md. Count it as reproduced if we land within 0.3 points of the paper's 8.75% error.

- Run `rb context --model <your model id>` first.
- Record the experiment, and each training setting from configs/train.yaml with the value the paper used, cited to its line in docs/baselines/resnet-paper.md (`--cited`, `--cited-source`).
- Record the paper's number as a criterion cited to its table row (`--source`, `--quote`, `--locator "Table 6"`), with my tolerance and my words.
- Run seeds 1 to 3 through `rb journal run`, and record each run.

Finish with `rb show` of the criterion exactly as printed, and every setting that differs from the paper's.

The paper’s text is saved in docs/baselines/resnet-paper.md:

docs/baselines/resnet-paper.md
# Deep Residual Learning for Image Recognition (2016): excerpts
Section 4.2, CIFAR-10:
We use a weight decay of 0.0001 and momentum of 0.9.
These models are trained with a mini-batch size of 128.
We start with a learning rate of 0.1.
Table 6, classification error on the CIFAR-10 test set:
| method | layers | params | error (%) |
|---|---|---|---|
| ResNet | 20 | 0.27M | 8.75 |
| ResNet | 32 | 0.46M | 7.51 |
Terminal window
$ rb show matches-paper-error
Criterion matches-paper-error on repro (primary, cited): Our ResNet-20 lands within 0.3 points of the paper's 8.75% error · words (as given): "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error."
cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75, read '| ResNet | 20 | 0.27M | 8.75 |'
matches-paper-error Our ResNet-20 lands within 0.3 points of the paper's 8.75% error
reproduced · measured · resnet20.error = 8.75 ± 0.3 (n ≥ 3)
reads: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb)
result: measured: resnet20 on its own, with its interval, so neither better nor worse
rule: reproduced: estimate 8.747 within 8.75 ± 0.3
timing: declared before the evidence (by production times)
units: 3 seeds
integrity: ok
rune5stgq: resnet20.error 8.81 (file) · unit seed=1 · seed=1
runx5twzo: resnet20.error 8.69 (file) · unit seed=2 · seed=2
runrvxoet: resnet20.error 8.74 (file) · unit seed=3 · seed=3
  1. Record the experiment’s purpose

    Terminal window
    $ rb experiment add "ResNet-20 against the ResNet paper (2016)" --id repro -q q-repro --purpose "Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6" --baseline resnet20
    Experiment repro: ResNet-20 against the ResNet paper (2016) · resnet20 (baseline) · under q-repro
    purpose: Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6
    Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
    Made 1 variant: resnet20 (baseline); a setting of one is rb spec set repro resnet20.<setting> <value>.
    Next: rb spec set repro --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e repro -q q-repro --metric <name> --at-most <x>
  2. Cite each setting and the paper’s value

    --cited keeps the paper’s value beside yours. A difference makes every criterion that cites the paper not comparable.

    Terminal window
    $ rb spec set repro optim.lr --source "configs/train.yaml#optim.lr" --cited 0.1 --cited-source docs/baselines/resnet-paper.md:7
    repro/optim.lr = 0.1 · verified in configs/train.yaml#optim.lr
    cited: 0.1 (docs/baselines/resnet-paper.md:7)
    $ rb spec set repro optim.momentum --source "configs/train.yaml#optim.momentum" --cited 0.9 --cited-source docs/baselines/resnet-paper.md:5
    repro/optim.momentum = 0.9 · verified in configs/train.yaml#optim.momentum
    cited: 0.9 (docs/baselines/resnet-paper.md:5)
    $ rb spec set repro optim.weight_decay --source "configs/train.yaml#optim.weight_decay" --cited 0.0001 --cited-source docs/baselines/resnet-paper.md:5
    repro/optim.weight_decay = 0.0001 · verified in configs/train.yaml#optim.weight_decay
    cited: 0.0001 (docs/baselines/resnet-paper.md:5)
    $ rb spec set repro data.batch_size --source "configs/train.yaml#data.batch_size" --cited 128 --cited-source docs/baselines/resnet-paper.md:6
    repro/data.batch_size = 128 · verified in configs/train.yaml#data.batch_size
    cited: 128 (docs/baselines/resnet-paper.md:6)
    $ rb spec set repro seed --per-run
    repro/seed · per-run: each run gives its own, with --set seed=<value> on rb run add
    Terminal window
    $ rb show repro/data.batch_size
    repro/data.batch_size = 128 · verified
    source: configs/train.yaml#data.batch_size · read 'data.batch_size: 128' @9d87bd151701
    cited source used 128 (docs/baselines/resnet-paper.md:6)
    history:
    2026-11-10T15:32:17 spec set: setting data.batch_size: new, 128
    2026-11-10T15:32:17 spec check: verified
  3. Cite the paper’s number as a criterion

    rb checks that the quoted row is in the file and states 8.75.

    Terminal window
    $ rb criterion add "Our ResNet-20 lands within 0.3 points of the paper's 8.75% error" -e repro -q q-repro --metric resnet20.error --equals 8.75 --tolerance 0.3 --source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6" --min-n 3 --words "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error." --id matches-paper-error
    matches-paper-error: Our ResNet-20 lands within 0.3 points of the paper's 8.75% error
    resnet20.error = 8.75 ± 0.3 (n ≥ 3) · primary · on repro
    cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75 (read '| ResNet | 20 | 0.27M | 8.75 |')
    declared; no evidence recorded yet
    Next: rb run add repro --from <the results file a run wrote> --set seed=<value>
  4. Run and record each seed

    Terminal window
    $ rb journal run -- python train.py --seed 1
    {"metrics": {"resnet20.error": 8.81}}
    exit 0. The command, its exit code and what it wrote are in the journal.
    It printed its metrics: resnet20.error=8.81. rb reconcile proposes the run of repro that records them.
    Since this session last looked:
    - out/seed1/ holds results no recorded experiment reads yet (rb reconcile)
    - out/seed1/metrics.json: results landed and are not in the record (rb reconcile)
    Next: rb reconcile
    Terminal window
    $ rb run add repro --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
    rune5stgq on repro: resnet20.error=8.81 · seed=1
    Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching.
    Produced: 2026-11-10 15:32:17 (file)
    matches-paper-error criterion undetermined · point only: resnet20.error: 8.81 % · mean, 1 seed (rb)
    rb put the code in git at refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30 removes it.
    Next: rb show matches-paper-error
    $ rb run add repro --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"
    runx5twzo on repro: resnet20.error=8.69 · seed=2
    Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching.
    Produced: 2026-11-10 15:32:17 (file)
    matches-paper-error criterion undetermined · point only: resnet20.error: 8.75 % · mean, 2 seeds (rb)
    Next: rb show matches-paper-error
    $ rb run add repro --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
    runrvxoet on repro: resnet20.error=8.74 · seed=3
    Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching.
    Produced: 2026-11-10 15:32:18 (file)
    matches-paper-error reproduced · measured: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb)
    Next: rb show matches-paper-error

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0