Declare the criterion before the runs
A threshold picked after the numbers came in was fitted to them. Declare the criterion, the rule a result must meet, and freeze it before the first run.
Paste this into your agent
Section titled “Paste this into your agent”Declare the criterion and freeze it before any run, then run the seeds.
Use Rabbit Brain to find out if warmup is worth it for the 6-layer GPT. docs/plan.md says what counts. - Run `rb context --model <your model id>` first. - Before any run: record the question, the experiment with both arms and every setting cited to its config, and the criterion docs/plan.md states, cited to its line. Then freeze the experiment. - Then run seeds 1 to 3 through `rb journal run -- python train.py --seed <N>`, and record each run from its output file, with its config. - Compute nothing yourself. Finish with `rb show` of the criterion exactly as printed.
What you’ll see
Section titled “What you’ll see”After three seeds:
$ rb show warmup-helpsCriterion warmup-helps on warmup (primary, own): Warmup lowers final validation loss by at least 0.02 nats warmup-helps Warmup lowers final validation loss by at least 0.02 nats criterion met · gain · change.val_loss ≤ -0.02 (n ≥ 3) reads: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (lower is better) rule: criterion met: estimate -0.02767 ≤ -0.02 timing: declared before the evidence (by production times) units: 3 seeds integrity: ok runttgab7: difference -0.031 (file) · unit seed=1 · seed=1 runvw3726: difference -0.026 (file) · unit seed=2 · seed=2 run2ugudg: difference -0.026 (file) · unit seed=3 · seed=3The timing line reads declared before the evidence.
Raising the bar to 0.03 nats now needs a reason, because the experiment is frozen:
$ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-barrb criterion add: Experiment warmup was frozen at 2026-11-10T15:32:13.795+00:00 (e84183e9a687); this changes it (new criterion stricter-warmup-bar: change.val_loss ≤ -0.03 (n ≥ 3)). → Give the change its reason: the same command with --amend --why "<reason>". The amendment is recorded, by whoever makes it.
$ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar --amend --why "a stricter bar, set after seeing the runs"stricter-warmup-bar: Warmup lowers final validation loss by at least 0.03 nats change.val_loss ≤ -0.03 (n ≥ 3) · primary · on warmup declared after the evidence existed: all 3 units were produced before it was declared it already reads change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb): criterion not met: estimate -0.02767 > -0.03
Next: rb run add warmup --from <the results file a run wrote> --set seed=<value>rb status keeps the two criteria apart by when each was declared:
$ rb statusWarmup on a small GPT · warmup-on-a-small-gpt14 record files not committed
Is warmup worth it for the 6-layer GPT? (open) On every warmup criterion below: units: 3 seeds. Warmup lowers final validation loss by at least 0.03 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion not met: estimate -0.02767 > -0.03; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [-0.03484, -0.0205] crosses -0.03: missed on the estimate, not across the whole interval. Warmup lowers final validation loss by at least 0.02 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion met: estimate -0.02767 ≤ -0.02; rule and question link declared before the evidence (by production times).
Rules: 1 met · 1 not met
Needs judgment (1) - stricter-warmup-bar misses its rule and no call on it is recorded: Warmup lowers final validation loss by at least 0.03 nats rb decide stricter-warmup-bar accept|reject|investigate --why "..."
The design of 1 experiment is not confirmed: rb confirm lists it.
Can be done now (1) - warmup: 1 criterion was declared after its evidence existed: a reading of its own needs its subject's runs made after the declaration rb run add warmup --from <the results file a run wrote> --set seed=<value> (run after the declaration)
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.rb checked the record with git at 2026-11-10 15:32 UTC: all 14 record files are new, changed or deleted since commit 90e2f71dfd90, which does not hold the record.Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/warmup/criteria/stricter-warmup-bar.yaml (new) and 11 moreTo check it yourself, in your terminal: git -C ~/warmup-gpt status --short --untracked-files=all -- .rb… (25 more lines)The late criterion reads declared after the evidence existed. Its miss waits for your call, a decision you record with rb decide. See Calls.
What rb did, step by step
Section titled “What rb did, step by step”-
Record the experiment and cite its settings
Terminal window $ rb experiment add "Warmup vs no warmup" --id warmup -q q-warmup --baseline none --candidate warmup --varies warmup_stepsExperiment warmup: Warmup vs no warmup · none (baseline) vs warmup (candidate) · varies warmup_steps · under q-warmupPlanned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.Made 2 variants: none (baseline), warmup (candidate); a setting of one is rb spec set warmup none.<setting> <value>.warmup declares it varies warmup_steps, but no variant has a value for it yet; give each variant its own: rb spec set warmup <variant>.warmup_steps <value> --source <file#key>Next: rb spec set warmup --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e warmup -q q-warmup --metric <name> --at-most <x>Terminal window $ rb spec set warmup --from configs/base.yaml --keys "optim.*,data.*"warmup: 4 settings from configs/base.yaml @90e2f71dfd90 · 4 verified$ rb spec set warmup none.warmup_steps 0 --source "configs/no-warmup.yaml#warmup_steps"warmup/none.warmup_steps = 0 · verified in configs/no-warmup.yaml#warmup_steps$ rb spec set warmup warmup.warmup_steps 500 --source "configs/warmup.yaml#warmup_steps"warmup/warmup.warmup_steps = 500 · verified in configs/warmup.yaml#warmup_steps$ rb spec set warmup seed --per-runwarmup/seed · per-run: each run gives its own, with --set seed=<value> on rb run add -
Declare the plan’s criterion
With
--why "docs/plan.md:3", rb checks that the line is there and states the threshold.Terminal window $ rb criterion add "Warmup lowers final validation loss by at least 0.02 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.02 --min-n 3 --why docs/plan.md:3 --id warmup-helpswarmup-helps: Warmup lowers final validation loss by at least 0.02 natschange.val_loss ≤ -0.02 (n ≥ 3) · primary · on warmupdeclared; no evidence recorded yetNext: rb run add warmup --from <the results file a run wrote> --set seed=<value> -
Freeze it
Terminal window $ rb freeze warmup -m "before the runs"warmup frozen: spec e84183e9a687 · before the runswarmup: Warmup vs no warmupvariants: none (baseline), warmup (candidate) · varies warmup_stepscriterion warmup-helps (primary): change.val_loss ≤ -0.02 (n ≥ 3)optim.lr = 0.0006 · verified · configs/base.yaml#optim.lroptim.weight_decay = 0.1 · verified · configs/base.yaml#optim.weight_decaydata.batch_size = 64 · verified · configs/base.yaml#data.batch_sizedata.seq_len = 256 · verified · configs/base.yaml#data.seq_lenseed = (each run) · per-run · no sourcenone.warmup_steps = 0 · verified · configs/no-warmup.yaml#warmup_stepswarmup.warmup_steps = 500 · verified · configs/warmup.yaml#warmup_stepslook: warmup-helps is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why "<reason>"; the amendment is kept with who made it.Next: rb run add warmup --from <metrics.json> --command "..." -
Run each seed
Terminal window $ rb journal run -- python train.py --seed 1seed 1: val_loss none 2.412 warmup 2.381{"metrics": {"none.val_loss": 2.412, "warmup.val_loss": 2.381}}exit 0. The command, its exit code and what it wrote are in the journal.It printed its metrics: none.val_loss=2.412, warmup.val_loss=2.381. rb reconcile proposes the run of warmup that records them.Since this session last looked:- out/seed1/ holds results no recorded experiment reads yet (rb reconcile)- out/seed1/metrics.json: results landed and are not in the record (rb reconcile)Next: rb reconcile -
Record each run
Terminal window $ rb run add warmup --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"runttgab7 on warmup: none.val_loss=2.412, warmup.val_loss=2.381 · seed=1Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching.Produced: 2026-11-10 15:32:13 (file)warmup-helps criterion undetermined · point only: change.val_loss: -0.031 nats · paired difference, 1 seed (rb)rb put the code in git at refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54 removes it.Next: rb show warmup-helps$ rb run add warmup --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"runvw3726 on warmup: none.val_loss=2.405, warmup.val_loss=2.379 · seed=2Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching.Produced: 2026-11-10 15:32:14 (file)warmup-helps criterion undetermined · point only: change.val_loss: -0.0285 nats · paired difference, 2 seeds (rb)Next: rb show warmup-helps$ rb run add warmup --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"run2ugudg on warmup: none.val_loss=2.418, warmup.val_loss=2.392 · seed=3Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching.Produced: 2026-11-10 15:32:14 (file)warmup-helps criterion met · gain: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb)Next: rb show warmup-helps
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0