Skip to content

Read a rule over seeds

“The mean meets 91.58” and “4 of 5 seeds meet 91.58” are different claims. rb checks the one your plan makes.

Paste this into your agent

Record the five seeds, and the plan's three rules over them.

Use Rabbit Brain to check INT8 against FP32 over the five seeds in results/int8-ptq/. docs/plan.md says what counts, one rule a line.

- Run `rb context --model <your model id>` first.
- Record the question and the experiment, with each setting cited to the config that states it.
- Record each of the plan's three rules as rb reads it, cited to its line: a count (`--count 4 --of 5`), a best of N (`--best --of 5`) and a relative change (`--relative`).
- Record each seed as a run.
- Compute nothing yourself: rb computes the findings.

Finish with `rb status` and `rb show` of each criterion exactly as printed.

The repository evaluates an FP32 ResNet-20 and its INT8 copy on CIFAR-10 for five seeds. Lines 4 to 6 of docs/plan.md hold the three rules.

rb status in your own terminal, once the agent is done:

Terminal window
$ rb status
INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10
Kept with rb since 2026-11-10. All 5 runs were recorded afterwards, not as they happened.
17 record files not committed
Does INT8 hold up against FP32 across the five seeds? (open)
On every int8-seeds criterion below: question link declared after the evidence existed: all 5 units were produced before it was declared; units: 5 seeds.
INT8's best of 5 seeds reaches 91.65% top-1. int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb); point only: a best of N is one unit's own value, with no interval; recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65; rule and question link declared after the evidence existed: all 5 units were produced before it was declared.
INT8 reaches 91.58% top-1 on at least 4 of 5 seeds. int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4; rule and question link declared after the evidence existed: all 5 units were produced before it was declared.
INT8 loses at most 0.1% of FP32's top-1. change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb); loss: the interval clear of 0 on the worse side (higher is better); estimate meets criterion: estimate -0.0916% ≥ -0.1%; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval.
Rules: 3 met, all 3 on the estimate alone
The design of 1 experiment is not confirmed: rb confirm lists it.
Facts that ask for nothing (1)
- int8-seeds: 3 criteria were declared after their evidence existed
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.
rb checked the record with git at 2026-11-10 15:32 UTC: all 17 record files are new, changed or deleted since commit 0e3550d9bdb6, which does not hold the record.
Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/int8-seeds/criteria/best-of-five.yaml (new) and 14 more
To check it yourself, in your terminal: git -C ~/int8-seeds status --short --untracked-files=all -- .rb
Rules
four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58
reads: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)
result: measured: int8 on its own, with its interval, so neither better nor worse
rule: recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4
timing: declared after the evidence existed: all 5 units were produced before it was declared
units: 5 seeds
best-of-five INT8's best of 5 seeds reaches 91.65% top-1
recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65
reads: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)
result: point only: a best of N is one unit's own value, with no interval
rule: recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65
timing: declared after the evidence existed: all 5 units were produced before it was declared
units: 5 seeds
relative-drop INT8 loses at most 0.1% of FP32's top-1
estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1%
reads: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)
result: loss: the interval clear of 0 on the worse side (higher is better)
rule: estimate meets criterion: estimate -0.0916% ≥ -0.1%
timing: declared after the evidence existed: all 5 units were produced before it was declared
units: 5 seeds
· the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval
Experiments
int8-seeds INT8 post-training quantization, five seeds · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 7 verified · 0 provisional · 0 unknown · 1 per-run · runs 5
to run again: code unknown · environment unknown · files ok
Runs: 5 recorded on 1 experiment · every receipt is what rb wrote
Next: rb confirm
  • A criterion is a rule a result must meet. Its id, such as four-of-five, is its name in every command.
  • A count or best of N that is met reads recorded values meet criterion: the recorded seeds themselves meet it.
  • The relative change reads estimate meets criterion: the estimate meets the rule, but its interval crosses -0.1.
  1. Declare the three rules

    Terminal window
    $ rb criterion add "INT8 reaches 91.58% top-1 on at least 4 of 5 seeds" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.58 --count 4 --of 5 --why docs/plan.md:4 --id four-of-five
    four-of-five: INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
    4 of 5 units' int8.top1 ≥ 91.58 · primary · on int8-seeds
    declared; no evidence recorded yet
    Next: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>
    $ rb criterion add "INT8's best of 5 seeds reaches 91.65% top-1" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.65 --best --of 5 --why docs/plan.md:5 --id best-of-five
    best-of-five: INT8's best of 5 seeds reaches 91.65% top-1
    the best of 5 units' int8.top1 ≥ 91.65 · primary · on int8-seeds
    declared; no evidence recorded yet
    Next: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>
    $ rb criterion add "INT8 loses at most 0.1% of FP32's top-1" -e int8-seeds -q q-seeds --metric change.top1 --at-least -0.1 --relative --why docs/plan.md:6 --id relative-drop
    relative-drop: INT8 loses at most 0.1% of FP32's top-1
    change.top1 as a relative change ≥ -0.1% · primary · on int8-seeds
    declared; no evidence recorded yet
    Next: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>
  2. Record each seed

    Terminal window
    $ rb run add int8-seeds --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml
    runbvd5mg on int8-seeds: fp32.top1=91.71, int8.top1=91.6 · seed=1
    Ran with the spec: 5 of its settings found in results/int8-ptq/seed1/config.yaml, all matching.
    Produced: 2026-10-20 10:00:00 (file)
    four-of-five criterion undetermined · point only: int8.top1: 91.6 % · mean, 1 seed (rb)
    best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 1 seed (rb)
    relative-drop criterion not met · point only: change.top1: -0.1199 % · relative change of the means, paired, 1 seed (rb)
    rb put the code in git at refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020 removes it.
    Since this session last looked:
    - four-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1 (rb decide four-of-five accept|reject|investigate --why "...")
    - best-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1 (rb decide best-of-five accept|reject|investigate --why "...")
    - relative-drop misses its rule and no call on it is recorded: INT8 loses at most 0.1% of FP32's top-1 (rb decide relative-drop accept|reject|investigate --why "...")
    - 4 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed2/, results/int8-ptq/seed3/, results/int8-ptq/seed4/ and 1 more (rb reconcile)
    Next: rb show four-of-five · rb show best-of-five · rb show relative-drop
    $ rb run add int8-seeds --from results/int8-ptq/seed2/metrics.json --set seed=2 --config results/int8-ptq/seed2/config.yaml
    run5zw7oj on int8-seeds: fp32.top1=91.66, int8.top1=91.58 · seed=2
    Ran with the spec: 5 of its settings found in results/int8-ptq/seed2/config.yaml, all matching.
    Produced: 2026-10-20 10:00:00 (file)
    four-of-five criterion undetermined · point only: int8.top1: 91.59 % · mean, 2 seeds (rb)
    best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 2 seeds (rb)
    relative-drop criterion not met · point only: change.top1: -0.1036 % · relative change of the means, paired, 2 seeds (rb)
    Since this session last looked:
    - 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed3/, results/int8-ptq/seed4/, results/int8-ptq/seed5/ (rb reconcile)
    Next: rb show four-of-five · rb show best-of-five · rb show relative-drop
    $ rb run add int8-seeds --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml
    runongfds on int8-seeds: fp32.top1=91.74, int8.top1=91.69 · seed=3
    Ran with the spec: 5 of its settings found in results/int8-ptq/seed3/config.yaml, all matching.
    Produced: 2026-10-20 10:00:00 (file)
    four-of-five criterion undetermined · measured: int8.top1: 91.62 [91.48, 91.77] % · mean, t interval, 3 seeds, 95% (rb)
    best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 3 seeds (rb)
    relative-drop estimate meets criterion · loss: change.top1: -0.08724 [-0.1685, -0.005936] % · relative change of the means, paired, delta-method t interval, 3 seeds, 95% (rb)
    Since this session last looked:
    - 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed4/, results/int8-ptq/seed5/ (rb reconcile)
    Next: rb show four-of-five · rb show best-of-five · rb show relative-drop
    $ rb run add int8-seeds --from results/int8-ptq/seed4/metrics.json --set seed=4 --config results/int8-ptq/seed4/config.yaml
    runmyetzp on int8-seeds: fp32.top1=91.69, int8.top1=91.57 · seed=4
    Ran with the spec: 5 of its settings found in results/int8-ptq/seed4/config.yaml, all matching.
    Produced: 2026-10-20 10:00:00 (file)
    four-of-five criterion undetermined · measured: int8.top1: 91.61 [91.52, 91.7] % · mean, t interval, 4 seeds, 95% (rb)
    best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 4 seeds (rb)
    relative-drop estimate meets criterion · loss: change.top1: -0.09815 [-0.153, -0.04325] % · relative change of the means, paired, delta-method t interval, 4 seeds, 95% (rb)
    Since this session last looked:
    - results/int8-ptq/seed5/ holds results written before the record began, which no recorded experiment reads yet (rb reconcile)
    Next: rb show four-of-five · rb show best-of-five · rb show relative-drop
    $ rb run add int8-seeds --from results/int8-ptq/seed5/metrics.json --set seed=5 --config results/int8-ptq/seed5/config.yaml
    runovg3mt on int8-seeds: fp32.top1=91.72, int8.top1=91.66 · seed=5
    Ran with the spec: 5 of its settings found in results/int8-ptq/seed5/config.yaml, all matching.
    Produced: 2026-10-20 10:00:00 (file)
    four-of-five recorded values meet criterion · measured: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)
    best-of-five recorded values meet criterion · point only: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)
    relative-drop estimate meets criterion · loss: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)
    Next: rb show four-of-five · rb show best-of-five · rb show relative-drop
  3. Check the file format

    These criteria are saved in format 2. rb 0.7.0 and 0.7.1 write nothing to a record with a format-2 file, so upgrade everyone who shares it together.

    Terminal window
    $ head -3 .rb/experiments/int8-seeds/criteria/four-of-five.yaml
    rb: 2
    kind: criterion
    id: four-of-five
  4. Read each criterion

    Terminal window
    $ rb show four-of-five
    Criterion four-of-five on int8-seeds (primary, own): INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
    four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds
    recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58
    reads: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)
    result: measured: int8 on its own, with its interval, so neither better nor worse
    rule: recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4
    timing: declared after the evidence existed: all 5 units were produced before it was declared
    units: 5 seeds
    integrity: ok
    runbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1
    run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2
    runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3
    runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4
    runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5
    Terminal window
    $ rb show best-of-five
    Criterion best-of-five on int8-seeds (primary, own): INT8's best of 5 seeds reaches 91.65% top-1
    best-of-five INT8's best of 5 seeds reaches 91.65% top-1
    recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65
    reads: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)
    result: point only: a best of N is one unit's own value, with no interval
    rule: recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65
    timing: declared after the evidence existed: all 5 units were produced before it was declared
    units: 5 seeds
    integrity: ok
    runbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1
    run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2
    runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3
    runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4
    runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5
    Terminal window
    $ rb show relative-drop
    Criterion relative-drop on int8-seeds (primary, own): INT8 loses at most 0.1% of FP32's top-1
    relative-drop INT8 loses at most 0.1% of FP32's top-1
    estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1%
    reads: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)
    result: loss: the interval clear of 0 on the worse side (higher is better)
    rule: estimate meets criterion: estimate -0.0916% ≥ -0.1%
    timing: declared after the evidence existed: all 5 units were produced before it was declared
    units: 5 seeds
    integrity: ok
    · the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval
    runbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1
    runbvd5mg: fp32.top1 91.71 (file) · unit seed=1 · produced before the declaration · seed=1
    run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2
    run5zw7oj: fp32.top1 91.66 (file) · unit seed=2 · produced before the declaration · seed=2
    runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3
    runongfds: fp32.top1 91.74 (file) · unit seed=3 · produced before the declaration · seed=3
    runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4
    runmyetzp: fp32.top1 91.69 (file) · unit seed=4 · produced before the declaration · seed=4
    runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5
    runovg3mt: fp32.top1 91.72 (file) · unit seed=5 · produced before the declaration · seed=5

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0