Get an interval, not a single number
With one seed you cannot tell a real cost from run-to-run noise. Ask for an interval in the criterion, the rule a result must meet, and set how many seeds it needs.
Paste this into your agent
Section titled “Paste this into your agent”Read the result on its 95% interval, over 5 seeds declared up front.
Use Rabbit Brain to find out whether dropout 0.1 costs us validation loss on the MLP. docs/plan.md says what counts. I want an interval, not one number: dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout. - Run `rb context --model <your model id>` first. - Before any run: record the experiment, its settings from the configs, the plan's criterion cited to its line, and my criterion on the interval, with my words, needing 5 seeds. Freeze it. - Run seeds 1 to 5 through `rb journal run`, and record each run. - After each seed, tell me rb's finding as printed. Never round "point only" up to a result. Finish with `rb show` of both criteria exactly as printed.
Two criteria read dropout’s validation loss minus no dropout’s:
mean-within-budgetis the plan’s: the mean cost is at most 0.02 nats.interval-within-budgetis yours: the whole 95% interval stays within 0.02 nats, on at least 5 seeds.
What you’ll see
Section titled “What you’ll see”After each seed, rb run add prints both findings, what rb reads for each criterion from the runs:
| Seeds | Reading | mean-within-budget | interval-within-budget |
|---|---|---|---|
| 1 | +0.008 nats | estimate meets criterion · point only | criterion undetermined · point only |
| 2 | +0.0025 nats | estimate meets criterion · point only | criterion undetermined · point only |
| 3 | +0.006333 [-0.01508, +0.02775] nats | estimate meets criterion · unresolved | criterion undetermined · unresolved |
| 4 | +0.0055 [-0.006011, +0.01701] nats | criterion met · unresolved | criterion undetermined · unresolved |
| 5 | +0.0056 [-0.002184, +0.01338] nats | criterion met · unresolved | criterion met · unresolved |
At three seeds the interval reaches past 0.02. Your criterion stays undetermined until its fifth seed.
$ rb statusDropout on a small MLP · dropout-on-a-small-mlp16 record files not committed
Does dropout 0.1 cost us validation loss on the MLP? (open) On every drop criterion below: question link declared before the evidence (by production times); units: 5 seeds. The whole 95% interval of dropout's cost stays within 0.02 nats. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: the interval's worse bound +0.01338 inside 0.02; rule and question link declared before the evidence (by production times) · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout". Dropout 0.1 costs at most 0.02 nats of validation loss. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: estimate +0.0056 ≤ 0.02; rule and question link declared before the evidence (by production times).
… (34 more lines)Unresolved means the interval includes 0. It is not a failure: both criteria ask only for a cost within 0.02 nats.
What rb did, step by step
Section titled “What rb did, step by step”-
Declare both criteria
Terminal window $ rb criterion add "Dropout 0.1 costs at most 0.02 nats of validation loss" -e drop -q q-drop --metric change.val_loss --at-most 0.02 --why docs/plan.md:3 --id mean-within-budgetmean-within-budget: Dropout 0.1 costs at most 0.02 nats of validation losschange.val_loss ≤ 0.02 · primary · on dropdeclared; no evidence recorded yetNext: rb run add drop --from <the results file a run wrote> --set seed=<value>$ rb criterion add "The whole 95% interval of dropout's cost stays within 0.02 nats" -e drop -q q-drop --metric change.val_loss --noninferior 0.02 --min-n 5 --words "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" --id interval-within-budgetinterval-within-budget: The whole 95% interval of dropout's cost stays within 0.02 natschange.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) · primary · on dropdeclared; no evidence recorded yetNext: rb run add drop --from <the results file a run wrote> --set seed=<value> -
Freeze the experiment
Terminal window $ rb freeze drop -m "before the first seed"drop frozen: spec 12f84c212b8c · before the first seeddrop: Dropout 0.1 vs nonevariants: none (baseline), p01 (candidate) · varies dropoutcriterion mean-within-budget (primary): change.val_loss ≤ 0.02criterion interval-within-budget (primary): change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) (words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout")optim.lr = 0.001 · verified · configs/train.yaml#optim.lrdata.batch_size = 256 · verified · configs/train.yaml#data.batch_sizeseed = (each run) · per-run · no sourcenone.dropout = 0.0 · verified · configs/none.yaml#dropoutp01.dropout = 0.1 · verified · configs/p01.yaml#dropoutlook: mean-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".look: interval-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why "<reason>"; the amendment is kept with who made it.Next: rb run add drop --from <metrics.json> --command "..." -
Run and record each seed
Terminal window $ rb journal run -- python train.py --seed 1{"metrics": {"none.val_loss": 1.31, "p01.val_loss": 1.318}}exit 0. The command, its exit code and what it wrote are in the journal.It printed its metrics: none.val_loss=1.31, p01.val_loss=1.318. rb reconcile proposes the run of drop that records them.Since this session last looked:- out/seed1/ holds results no recorded experiment reads yet (rb reconcile)- out/seed1/metrics.json: results landed and are not in the record (rb reconcile)Next: rb reconcileTerminal window $ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"rungxyf6p on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching.Produced: 2026-11-10 15:32:22 (file)mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it.Next: rb show mean-within-budget · rb show interval-within-budget$ rb run add drop --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"runbklxmd on drop: none.val_loss=1.302, p01.val_loss=1.299 · seed=2Ran with the spec: 3 of its settings found in out/seed2/config.yaml, all matching.Produced: 2026-11-10 15:32:23 (file)mean-within-budget estimate meets criterion · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb)interval-within-budget criterion undetermined · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb)Next: rb show mean-within-budget · rb show interval-within-budget$ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"runj5w6lb on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching.Produced: 2026-11-10 15:32:23 (file)mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)Next: rb show mean-within-budget · rb show interval-within-budget$ rb run add drop --from out/seed4/metrics.json --set seed=4 --config out/seed4/config.yaml --command "python train.py --seed 4"runrevodj on drop: none.val_loss=1.308, p01.val_loss=1.311 · seed=4Ran with the spec: 3 of its settings found in out/seed4/config.yaml, all matching.Produced: 2026-11-10 15:32:24 (file)mean-within-budget criterion met · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb)interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb)Next: rb show mean-within-budget · rb show interval-within-budget$ rb run add drop --from out/seed5/metrics.json --set seed=5 --config out/seed5/config.yaml --command "python train.py --seed 5"rungtlduu on drop: none.val_loss=1.312, p01.val_loss=1.318 · seed=5Ran with the spec: 3 of its settings found in out/seed5/config.yaml, all matching.Produced: 2026-11-10 15:32:24 (file)mean-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)interval-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)Next: rb show mean-within-budget · rb show interval-within-budget -
Read your criterion
Terminal window $ rb show interval-within-budgetCriterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout"interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 natscriterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5)reads: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)result: unresolved: the interval includes 0rule: criterion met: the interval's worse bound +0.01338 inside 0.02timing: declared before the evidence (by production times)units: 5 seedsintegrity: okrungxyf6p: difference +0.008 (file) · unit seed=1 · seed=1runbklxmd: difference -0.003 (file) · unit seed=2 · seed=2runj5w6lb: difference +0.014 (file) · unit seed=3 · seed=3runrevodj: difference +0.003 (file) · unit seed=4 · seed=4rungtlduu: difference +0.006 (file) · unit seed=5 · seed=5 -
Compare the arms
Terminal window $ rb compare dropdrop Dropout 0.1 vs none · varies dropout · 5 runs under the current spec · means over seedsvariant role dropout val_loss (nats)none baseline 0.0 1.309 (n=5)p01 candidate 0.1 1.315 +0.0056 worse [primary] (n=5)the primary criteria read: val_loss p01 vs none (mean-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)), val_loss p01 vs none (interval-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)); deltas are means, the findings carry the intervals (rb show <criterion>)
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0