Findings
A finding is rb’s reading of a criterion, a rule a result must meet, against the runs. rb computes it on every read and never stores it. rb show after five seeds:
$ rb show interval-within-budgetCriterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) reads: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb) result: unresolved: the interval includes 0 rule: criterion met: the interval's worse bound +0.01338 inside 0.02 timing: declared before the evidence (by production times) units: 5 seeds integrity: ok rungxyf6p: difference +0.008 (file) · unit seed=1 · seed=1 runbklxmd: difference -0.003 (file) · unit seed=2 · seed=2 runj5w6lb: difference +0.014 (file) · unit seed=3 · seed=3 runrevodj: difference +0.003 (file) · unit seed=4 · seed=4 rungtlduu: difference +0.006 (file) · unit seed=5 · seed=5What each line says
Section titled “What each line says”| Line | What it says |
|---|---|
reads | The number rb read, its interval and how it was computed. (rb) means rb computed it from the runs. |
result | What the number shows. gain or loss: the interval is clear of 0 on the better or worse side. unresolved: it includes 0. equivalent: inside an --equivalent margin. point only: no interval yet. measured: one variant’s number on its own. |
rule | Whether the rule is met. criterion undetermined: it needs an interval, more units or the metric’s direction. estimate meets criterion: it holds, but with no interval or one that crosses the threshold. |
timing | Whether the criterion was declared before its evidence. See Criteria. |
units | How many independent units the reading rests on, here seeds. |
integrity | Problems with what the finding depends on, or ok. |
Rule and result are separate: above, the rule is met while the result is unresolved.
How rb reads the runs
Section titled “How rb reads the runs”- Each run gives one difference between the two arms compared: the subject’s number minus the reference’s.
- Runs with the same per-run values (two runs at
seed=2) are one unit, their numbers averaged. - The estimate is the mean over units. From 3 units, rb gives a 95% Student t interval.
- Arms recorded in separate runs pair up by unit (“paired by seed”). Arms that never share a unit are compared unpaired (Welch).
- Declare a setting that only numbers repeats, such as an epoch, with
rb spec set <exp> population.repeat epoch --no-verify(orseed,epoch). rb then never offers it as the unit.
As seeds come in
Section titled “As seeds come in”With one or two seeds there is no interval, so estimate meets criterion does not mean it works. Two criteria, one on the estimate and one on the interval, after the first seed:
$ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"rungxyf6p on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1 Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching. Produced: 2026-11-10 15:32:22 (file) mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb) interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb) rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it.
Next: rb show mean-within-budget · rb show interval-within-budgetAfter the third, the interval appears but still reaches past 0.02, and the second criterion asks for 5 seeds:
$ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"runj5w6lb on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3 Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching. Produced: 2026-11-10 15:32:23 (file) mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb) interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
Next: rb show mean-within-budget · rb show interval-within-budgetGet an interval, not a single number walks through all five seeds.
Integrity
Section titled “Integrity”The common problems:
| Problem | What it means |
|---|---|
confound | the arms differ in a setting the experiment does not say it varies, and nobody accepted it |
conflict, stale | a setting’s source states another value, or no longer states it |
unknown | a required setting nobody has found |
not_comparable | a setting differs from the value a cited source used, or rb’s paired reading cannot give a cited difference of means |
drift | a frozen experiment no longer matches its freeze |
edited_outside_rb | a record it depends on was changed by hand |
A provisional setting (given but not checked), numbers typed in, a run whose config says the code had uncommitted changes, and arms that differ in a per-run setting are notes on a finding, not problems.
rb status --fail-on <gate> exits 1 when the record fails that gate. Most CI jobs want unestablished. It fails on a primary criterion that is not met, undetermined, untested or not shown to be declared before what it reads, on any integrity problem, and on a record changed outside rb.
rb status --fail-on unestablishedCheck out the whole history in CI (fetch-depth: 0 with actions/checkout). rb tells a clean merge from a hand edit only through git’s history. rb docs says what each gate checks.
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0