Skip to content

Declare the criterion before the runs

A threshold picked after the numbers came in was fitted to them. Declare the criterion, the rule a result must meet, and freeze it before the first run.

Paste this into your agent

Declare the criterion and freeze it before any run, then run the seeds.

Use Rabbit Brain to find out if warmup is worth it for the 6-layer GPT. docs/plan.md says what counts.

- Run `rb context --model <your model id>` first.
- Before any run: record the question, the experiment with both arms and every setting cited to its config, and the criterion docs/plan.md states, cited to its line. Then freeze the experiment.
- Then run seeds 1 to 3 through `rb journal run -- python train.py --seed <N>`, and record each run from its output file, with its config.
- Compute nothing yourself.

Finish with `rb show` of the criterion exactly as printed.

After three seeds:

Terminal window
$ rb show warmup-helps
Criterion warmup-helps on warmup (primary, own): Warmup lowers final validation loss by at least 0.02 nats
warmup-helps Warmup lowers final validation loss by at least 0.02 nats
criterion met · gain · change.val_loss ≤ -0.02 (n ≥ 3)
reads: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb)
result: gain: the interval clear of 0 on the better side (lower is better)
rule: criterion met: estimate -0.02767 ≤ -0.02
timing: declared before the evidence (by production times)
units: 3 seeds
integrity: ok
runttgab7: difference -0.031 (file) · unit seed=1 · seed=1
runvw3726: difference -0.026 (file) · unit seed=2 · seed=2
run2ugudg: difference -0.026 (file) · unit seed=3 · seed=3

The timing line reads declared before the evidence.

Raising the bar to 0.03 nats now needs a reason, because the experiment is frozen:

Terminal window
$ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar
rb criterion add: Experiment warmup was frozen at 2026-11-10T15:32:13.795+00:00 (e84183e9a687); this changes it (new criterion stricter-warmup-bar: change.val_loss ≤ -0.03 (n ≥ 3)).
→ Give the change its reason: the same command with --amend --why "<reason>". The amendment is recorded, by whoever makes it.
$ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar --amend --why "a stricter bar, set after seeing the runs"
stricter-warmup-bar: Warmup lowers final validation loss by at least 0.03 nats
change.val_loss ≤ -0.03 (n ≥ 3) · primary · on warmup
declared after the evidence existed: all 3 units were produced before it was declared
it already reads change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb): criterion not met: estimate -0.02767 > -0.03
Next: rb run add warmup --from <the results file a run wrote> --set seed=<value>

rb status keeps the two criteria apart by when each was declared:

Terminal window
$ rb status
Warmup on a small GPT · warmup-on-a-small-gpt
14 record files not committed
Is warmup worth it for the 6-layer GPT? (open)
On every warmup criterion below: units: 3 seeds.
Warmup lowers final validation loss by at least 0.03 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion not met: estimate -0.02767 > -0.03; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [-0.03484, -0.0205] crosses -0.03: missed on the estimate, not across the whole interval.
Warmup lowers final validation loss by at least 0.02 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion met: estimate -0.02767 ≤ -0.02; rule and question link declared before the evidence (by production times).
Rules: 1 met · 1 not met
Needs judgment (1)
- stricter-warmup-bar misses its rule and no call on it is recorded: Warmup lowers final validation loss by at least 0.03 nats
rb decide stricter-warmup-bar accept|reject|investigate --why "..."
The design of 1 experiment is not confirmed: rb confirm lists it.
Can be done now (1)
- warmup: 1 criterion was declared after its evidence existed: a reading of its own needs its subject's runs made after the declaration
rb run add warmup --from <the results file a run wrote> --set seed=<value> (run after the declaration)
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.
rb checked the record with git at 2026-11-10 15:32 UTC: all 14 record files are new, changed or deleted since commit 90e2f71dfd90, which does not hold the record.
Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/warmup/criteria/stricter-warmup-bar.yaml (new) and 11 more
To check it yourself, in your terminal: git -C ~/warmup-gpt status --short --untracked-files=all -- .rb
… (25 more lines)

The late criterion reads declared after the evidence existed. Its miss waits for your call, a decision you record with rb decide. See Calls.

  1. Record the experiment and cite its settings

    Terminal window
    $ rb experiment add "Warmup vs no warmup" --id warmup -q q-warmup --baseline none --candidate warmup --varies warmup_steps
    Experiment warmup: Warmup vs no warmup · none (baseline) vs warmup (candidate) · varies warmup_steps · under q-warmup
    Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
    Made 2 variants: none (baseline), warmup (candidate); a setting of one is rb spec set warmup none.<setting> <value>.
    warmup declares it varies warmup_steps, but no variant has a value for it yet; give each variant its own: rb spec set warmup <variant>.warmup_steps <value> --source <file#key>
    Next: rb spec set warmup --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e warmup -q q-warmup --metric <name> --at-most <x>
    Terminal window
    $ rb spec set warmup --from configs/base.yaml --keys "optim.*,data.*"
    warmup: 4 settings from configs/base.yaml @90e2f71dfd90 · 4 verified
    $ rb spec set warmup none.warmup_steps 0 --source "configs/no-warmup.yaml#warmup_steps"
    warmup/none.warmup_steps = 0 · verified in configs/no-warmup.yaml#warmup_steps
    $ rb spec set warmup warmup.warmup_steps 500 --source "configs/warmup.yaml#warmup_steps"
    warmup/warmup.warmup_steps = 500 · verified in configs/warmup.yaml#warmup_steps
    $ rb spec set warmup seed --per-run
    warmup/seed · per-run: each run gives its own, with --set seed=<value> on rb run add
  2. Declare the plan’s criterion

    With --why "docs/plan.md:3", rb checks that the line is there and states the threshold.

    Terminal window
    $ rb criterion add "Warmup lowers final validation loss by at least 0.02 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.02 --min-n 3 --why docs/plan.md:3 --id warmup-helps
    warmup-helps: Warmup lowers final validation loss by at least 0.02 nats
    change.val_loss ≤ -0.02 (n ≥ 3) · primary · on warmup
    declared; no evidence recorded yet
    Next: rb run add warmup --from <the results file a run wrote> --set seed=<value>
  3. Freeze it

    Terminal window
    $ rb freeze warmup -m "before the runs"
    warmup frozen: spec e84183e9a687 · before the runs
    warmup: Warmup vs no warmup
    variants: none (baseline), warmup (candidate) · varies warmup_steps
    criterion warmup-helps (primary): change.val_loss ≤ -0.02 (n ≥ 3)
    optim.lr = 0.0006 · verified · configs/base.yaml#optim.lr
    optim.weight_decay = 0.1 · verified · configs/base.yaml#optim.weight_decay
    data.batch_size = 64 · verified · configs/base.yaml#data.batch_size
    data.seq_len = 256 · verified · configs/base.yaml#data.seq_len
    seed = (each run) · per-run · no source
    none.warmup_steps = 0 · verified · configs/no-warmup.yaml#warmup_steps
    warmup.warmup_steps = 500 · verified · configs/warmup.yaml#warmup_steps
    look: warmup-helps is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".
    Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why "<reason>"; the amendment is kept with who made it.
    Next: rb run add warmup --from <metrics.json> --command "..."
  4. Run each seed

    Terminal window
    $ rb journal run -- python train.py --seed 1
    seed 1: val_loss none 2.412 warmup 2.381
    {"metrics": {"none.val_loss": 2.412, "warmup.val_loss": 2.381}}
    exit 0. The command, its exit code and what it wrote are in the journal.
    It printed its metrics: none.val_loss=2.412, warmup.val_loss=2.381. rb reconcile proposes the run of warmup that records them.
    Since this session last looked:
    - out/seed1/ holds results no recorded experiment reads yet (rb reconcile)
    - out/seed1/metrics.json: results landed and are not in the record (rb reconcile)
    Next: rb reconcile
  5. Record each run

    Terminal window
    $ rb run add warmup --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
    runttgab7 on warmup: none.val_loss=2.412, warmup.val_loss=2.381 · seed=1
    Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching.
    Produced: 2026-11-10 15:32:13 (file)
    warmup-helps criterion undetermined · point only: change.val_loss: -0.031 nats · paired difference, 1 seed (rb)
    rb put the code in git at refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54 removes it.
    Next: rb show warmup-helps
    $ rb run add warmup --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"
    runvw3726 on warmup: none.val_loss=2.405, warmup.val_loss=2.379 · seed=2
    Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching.
    Produced: 2026-11-10 15:32:14 (file)
    warmup-helps criterion undetermined · point only: change.val_loss: -0.0285 nats · paired difference, 2 seeds (rb)
    Next: rb show warmup-helps
    $ rb run add warmup --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
    run2ugudg on warmup: none.val_loss=2.418, warmup.val_loss=2.392 · seed=3
    Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching.
    Produced: 2026-11-10 15:32:14 (file)
    warmup-helps criterion met · gain: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb)
    Next: rb show warmup-helps

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0