Skip to content

Get an interval, not a single number

With one seed you cannot tell a real cost from run-to-run noise. Ask for an interval in the criterion, the rule a result must meet, and set how many seeds it needs.

Paste this into your agent

Read the result on its 95% interval, over 5 seeds declared up front.

Use Rabbit Brain to find out whether dropout 0.1 costs us validation loss on the MLP. docs/plan.md says what counts. I want an interval, not one number: dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout.

- Run `rb context --model <your model id>` first.
- Before any run: record the experiment, its settings from the configs, the plan's criterion cited to its line, and my criterion on the interval, with my words, needing 5 seeds. Freeze it.
- Run seeds 1 to 5 through `rb journal run`, and record each run.
- After each seed, tell me rb's finding as printed. Never round "point only" up to a result.

Finish with `rb show` of both criteria exactly as printed.

Two criteria read dropout’s validation loss minus no dropout’s:

  • mean-within-budget is the plan’s: the mean cost is at most 0.02 nats.
  • interval-within-budget is yours: the whole 95% interval stays within 0.02 nats, on at least 5 seeds.

After each seed, rb run add prints both findings, what rb reads for each criterion from the runs:

SeedsReadingmean-within-budgetinterval-within-budget
1+0.008 natsestimate meets criterion · point onlycriterion undetermined · point only
2+0.0025 natsestimate meets criterion · point onlycriterion undetermined · point only
3+0.006333 [-0.01508, +0.02775] natsestimate meets criterion · unresolvedcriterion undetermined · unresolved
4+0.0055 [-0.006011, +0.01701] natscriterion met · unresolvedcriterion undetermined · unresolved
5+0.0056 [-0.002184, +0.01338] natscriterion met · unresolvedcriterion met · unresolved

At three seeds the interval reaches past 0.02. Your criterion stays undetermined until its fifth seed.

Terminal window
$ rb status
Dropout on a small MLP · dropout-on-a-small-mlp
16 record files not committed
Does dropout 0.1 cost us validation loss on the MLP? (open)
On every drop criterion below: question link declared before the evidence (by production times); units: 5 seeds.
The whole 95% interval of dropout's cost stays within 0.02 nats. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: the interval's worse bound +0.01338 inside 0.02; rule and question link declared before the evidence (by production times) · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout".
Dropout 0.1 costs at most 0.02 nats of validation loss. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: estimate +0.0056 ≤ 0.02; rule and question link declared before the evidence (by production times).
… (34 more lines)

Unresolved means the interval includes 0. It is not a failure: both criteria ask only for a cost within 0.02 nats.

  1. Declare both criteria

    Terminal window
    $ rb criterion add "Dropout 0.1 costs at most 0.02 nats of validation loss" -e drop -q q-drop --metric change.val_loss --at-most 0.02 --why docs/plan.md:3 --id mean-within-budget
    mean-within-budget: Dropout 0.1 costs at most 0.02 nats of validation loss
    change.val_loss ≤ 0.02 · primary · on drop
    declared; no evidence recorded yet
    Next: rb run add drop --from <the results file a run wrote> --set seed=<value>
    $ rb criterion add "The whole 95% interval of dropout's cost stays within 0.02 nats" -e drop -q q-drop --metric change.val_loss --noninferior 0.02 --min-n 5 --words "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" --id interval-within-budget
    interval-within-budget: The whole 95% interval of dropout's cost stays within 0.02 nats
    change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) · primary · on drop
    declared; no evidence recorded yet
    Next: rb run add drop --from <the results file a run wrote> --set seed=<value>
  2. Freeze the experiment

    Terminal window
    $ rb freeze drop -m "before the first seed"
    drop frozen: spec 12f84c212b8c · before the first seed
    drop: Dropout 0.1 vs none
    variants: none (baseline), p01 (candidate) · varies dropout
    criterion mean-within-budget (primary): change.val_loss ≤ 0.02
    criterion interval-within-budget (primary): change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) (words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout")
    optim.lr = 0.001 · verified · configs/train.yaml#optim.lr
    data.batch_size = 256 · verified · configs/train.yaml#data.batch_size
    seed = (each run) · per-run · no source
    none.dropout = 0.0 · verified · configs/none.yaml#dropout
    p01.dropout = 0.1 · verified · configs/p01.yaml#dropout
    look: mean-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".
    look: interval-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".
    Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why "<reason>"; the amendment is kept with who made it.
    Next: rb run add drop --from <metrics.json> --command "..."
  3. Run and record each seed

    Terminal window
    $ rb journal run -- python train.py --seed 1
    {"metrics": {"none.val_loss": 1.31, "p01.val_loss": 1.318}}
    exit 0. The command, its exit code and what it wrote are in the journal.
    It printed its metrics: none.val_loss=1.31, p01.val_loss=1.318. rb reconcile proposes the run of drop that records them.
    Since this session last looked:
    - out/seed1/ holds results no recorded experiment reads yet (rb reconcile)
    - out/seed1/metrics.json: results landed and are not in the record (rb reconcile)
    Next: rb reconcile
    Terminal window
    $ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
    rungxyf6p on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1
    Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching.
    Produced: 2026-11-10 15:32:22 (file)
    mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
    interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
    rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it.
    Next: rb show mean-within-budget · rb show interval-within-budget
    $ rb run add drop --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"
    runbklxmd on drop: none.val_loss=1.302, p01.val_loss=1.299 · seed=2
    Ran with the spec: 3 of its settings found in out/seed2/config.yaml, all matching.
    Produced: 2026-11-10 15:32:23 (file)
    mean-within-budget estimate meets criterion · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb)
    interval-within-budget criterion undetermined · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb)
    Next: rb show mean-within-budget · rb show interval-within-budget
    $ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
    runj5w6lb on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3
    Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching.
    Produced: 2026-11-10 15:32:23 (file)
    mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
    interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
    Next: rb show mean-within-budget · rb show interval-within-budget
    $ rb run add drop --from out/seed4/metrics.json --set seed=4 --config out/seed4/config.yaml --command "python train.py --seed 4"
    runrevodj on drop: none.val_loss=1.308, p01.val_loss=1.311 · seed=4
    Ran with the spec: 3 of its settings found in out/seed4/config.yaml, all matching.
    Produced: 2026-11-10 15:32:24 (file)
    mean-within-budget criterion met · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb)
    interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb)
    Next: rb show mean-within-budget · rb show interval-within-budget
    $ rb run add drop --from out/seed5/metrics.json --set seed=5 --config out/seed5/config.yaml --command "python train.py --seed 5"
    rungtlduu on drop: none.val_loss=1.312, p01.val_loss=1.318 · seed=5
    Ran with the spec: 3 of its settings found in out/seed5/config.yaml, all matching.
    Produced: 2026-11-10 15:32:24 (file)
    mean-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
    interval-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
    Next: rb show mean-within-budget · rb show interval-within-budget
  4. Read your criterion

    Terminal window
    $ rb show interval-within-budget
    Criterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout"
    interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats
    criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5)
    reads: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
    result: unresolved: the interval includes 0
    rule: criterion met: the interval's worse bound +0.01338 inside 0.02
    timing: declared before the evidence (by production times)
    units: 5 seeds
    integrity: ok
    rungxyf6p: difference +0.008 (file) · unit seed=1 · seed=1
    runbklxmd: difference -0.003 (file) · unit seed=2 · seed=2
    runj5w6lb: difference +0.014 (file) · unit seed=3 · seed=3
    runrevodj: difference +0.003 (file) · unit seed=4 · seed=4
    rungtlduu: difference +0.006 (file) · unit seed=5 · seed=5
  5. Compare the arms

    Terminal window
    $ rb compare drop
    drop Dropout 0.1 vs none · varies dropout · 5 runs under the current spec · means over seeds
    variant role dropout val_loss (nats)
    none baseline 0.0 1.309 (n=5)
    p01 candidate 0.1 1.315 +0.0056 worse [primary] (n=5)
    the primary criteria read: val_loss p01 vs none (mean-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)), val_loss p01 vs none (interval-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)); deltas are means, the findings carry the intervals (rb show <criterion>)

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0