Skip to content

Findings

A finding is rb’s reading of a criterion, a rule a result must meet, against the runs. rb computes it on every read and never stores it. rb show after five seeds:

Terminal window
$ rb show interval-within-budget
Criterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout"
interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats
criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5)
reads: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
result: unresolved: the interval includes 0
rule: criterion met: the interval's worse bound +0.01338 inside 0.02
timing: declared before the evidence (by production times)
units: 5 seeds
integrity: ok
rungxyf6p: difference +0.008 (file) · unit seed=1 · seed=1
runbklxmd: difference -0.003 (file) · unit seed=2 · seed=2
runj5w6lb: difference +0.014 (file) · unit seed=3 · seed=3
runrevodj: difference +0.003 (file) · unit seed=4 · seed=4
rungtlduu: difference +0.006 (file) · unit seed=5 · seed=5
LineWhat it says
readsThe number rb read, its interval and how it was computed. (rb) means rb computed it from the runs.
resultWhat the number shows. gain or loss: the interval is clear of 0 on the better or worse side. unresolved: it includes 0. equivalent: inside an --equivalent margin. point only: no interval yet. measured: one variant’s number on its own.
ruleWhether the rule is met. criterion undetermined: it needs an interval, more units or the metric’s direction. estimate meets criterion: it holds, but with no interval or one that crosses the threshold.
timingWhether the criterion was declared before its evidence. See Criteria.
unitsHow many independent units the reading rests on, here seeds.
integrityProblems with what the finding depends on, or ok.

Rule and result are separate: above, the rule is met while the result is unresolved.

  • Each run gives one difference between the two arms compared: the subject’s number minus the reference’s.
  • Runs with the same per-run values (two runs at seed=2) are one unit, their numbers averaged.
  • The estimate is the mean over units. From 3 units, rb gives a 95% Student t interval.
  • Arms recorded in separate runs pair up by unit (“paired by seed”). Arms that never share a unit are compared unpaired (Welch).
  • Declare a setting that only numbers repeats, such as an epoch, with rb spec set <exp> population.repeat epoch --no-verify (or seed,epoch). rb then never offers it as the unit.

With one or two seeds there is no interval, so estimate meets criterion does not mean it works. Two criteria, one on the estimate and one on the interval, after the first seed:

Terminal window
$ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
rungxyf6p on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1
Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching.
Produced: 2026-11-10 15:32:22 (file)
mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it.
Next: rb show mean-within-budget · rb show interval-within-budget

After the third, the interval appears but still reaches past 0.02, and the second criterion asks for 5 seeds:

Terminal window
$ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
runj5w6lb on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3
Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching.
Produced: 2026-11-10 15:32:23 (file)
mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
Next: rb show mean-within-budget · rb show interval-within-budget

Get an interval, not a single number walks through all five seeds.

The common problems:

ProblemWhat it means
confoundthe arms differ in a setting the experiment does not say it varies, and nobody accepted it
conflict, stalea setting’s source states another value, or no longer states it
unknowna required setting nobody has found
not_comparablea setting differs from the value a cited source used, or rb’s paired reading cannot give a cited difference of means
drifta frozen experiment no longer matches its freeze
edited_outside_rba record it depends on was changed by hand

A provisional setting (given but not checked), numbers typed in, a run whose config says the code had uncommitted changes, and arms that differ in a per-run setting are notes on a finding, not problems.

rb status --fail-on <gate> exits 1 when the record fails that gate. Most CI jobs want unestablished. It fails on a primary criterion that is not met, undetermined, untested or not shown to be declared before what it reads, on any integrity problem, and on a record changed outside rb.

Terminal window
rb status --fail-on unestablished

Check out the whole history in CI (fetch-depth: 0 with actions/checkout). rb tells a clean merge from a hand edit only through git’s history. rb docs says what each gate checks.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0