Read a rule over seeds
“The mean meets 91.58” and “4 of 5 seeds meet 91.58” are different claims. rb checks the one your plan makes.
Paste this into your agent
Section titled “Paste this into your agent”Record the five seeds, and the plan's three rules over them.
Use Rabbit Brain to check INT8 against FP32 over the five seeds in results/int8-ptq/. docs/plan.md says what counts, one rule a line. - Run `rb context --model <your model id>` first. - Record the question and the experiment, with each setting cited to the config that states it. - Record each of the plan's three rules as rb reads it, cited to its line: a count (`--count 4 --of 5`), a best of N (`--best --of 5`) and a relative change (`--relative`). - Record each seed as a run. - Compute nothing yourself: rb computes the findings. Finish with `rb status` and `rb show` of each criterion exactly as printed.
The repository evaluates an FP32 ResNet-20 and its INT8 copy on CIFAR-10 for five seeds. Lines 4 to 6 of docs/plan.md hold the three rules.
What you’ll see
Section titled “What you’ll see”rb status in your own terminal, once the agent is done:
$ rb statusINT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10Kept with rb since 2026-11-10. All 5 runs were recorded afterwards, not as they happened.17 record files not committed
Does INT8 hold up against FP32 across the five seeds? (open) On every int8-seeds criterion below: question link declared after the evidence existed: all 5 units were produced before it was declared; units: 5 seeds. INT8's best of 5 seeds reaches 91.65% top-1. int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb); point only: a best of N is one unit's own value, with no interval; recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. INT8 reaches 91.58% top-1 on at least 4 of 5 seeds. int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. INT8 loses at most 0.1% of FP32's top-1. change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb); loss: the interval clear of 0 on the worse side (higher is better); estimate meets criterion: estimate -0.0916% ≥ -0.1%; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval.
Rules: 3 met, all 3 on the estimate alone
The design of 1 experiment is not confirmed: rb confirm lists it.
Facts that ask for nothing (1) - int8-seeds: 3 criteria were declared after their evidence existed
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.rb checked the record with git at 2026-11-10 15:32 UTC: all 17 record files are new, changed or deleted since commit 0e3550d9bdb6, which does not hold the record.Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/int8-seeds/criteria/best-of-five.yaml (new) and 14 moreTo check it yourself, in your terminal: git -C ~/int8-seeds status --short --untracked-files=all -- .rb
Rules four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58 reads: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb) result: measured: int8 on its own, with its interval, so neither better nor worse rule: recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4 timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds best-of-five INT8's best of 5 seeds reaches 91.65% top-1 recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65 reads: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb) result: point only: a best of N is one unit's own value, with no interval rule: recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65 timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds relative-drop INT8 loses at most 0.1% of FP32's top-1 estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1% reads: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: estimate meets criterion: estimate -0.0916% ≥ -0.1% timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds · the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval
Experiments int8-seeds INT8 post-training quantization, five seeds · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 7 verified · 0 provisional · 0 unknown · 1 per-run · runs 5 to run again: code unknown · environment unknown · files ok
Runs: 5 recorded on 1 experiment · every receipt is what rb wrote
Next: rb confirm- A criterion is a rule a result must meet. Its id, such as
four-of-five, is its name in every command. - A count or best of N that is met reads recorded values meet criterion: the recorded seeds themselves meet it.
- The relative change reads estimate meets criterion: the estimate meets the rule, but its interval crosses -0.1.
What rb did, step by step
Section titled “What rb did, step by step”-
Declare the three rules
Terminal window $ rb criterion add "INT8 reaches 91.58% top-1 on at least 4 of 5 seeds" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.58 --count 4 --of 5 --why docs/plan.md:4 --id four-of-fivefour-of-five: INT8 reaches 91.58% top-1 on at least 4 of 5 seeds4 of 5 units' int8.top1 ≥ 91.58 · primary · on int8-seedsdeclared; no evidence recorded yetNext: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>$ rb criterion add "INT8's best of 5 seeds reaches 91.65% top-1" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.65 --best --of 5 --why docs/plan.md:5 --id best-of-fivebest-of-five: INT8's best of 5 seeds reaches 91.65% top-1the best of 5 units' int8.top1 ≥ 91.65 · primary · on int8-seedsdeclared; no evidence recorded yetNext: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value>$ rb criterion add "INT8 loses at most 0.1% of FP32's top-1" -e int8-seeds -q q-seeds --metric change.top1 --at-least -0.1 --relative --why docs/plan.md:6 --id relative-droprelative-drop: INT8 loses at most 0.1% of FP32's top-1change.top1 as a relative change ≥ -0.1% · primary · on int8-seedsdeclared; no evidence recorded yetNext: rb run add int8-seeds --from <the results file a run wrote> --set seed=<value> -
Record each seed
Terminal window $ rb run add int8-seeds --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yamlrunbvd5mg on int8-seeds: fp32.top1=91.71, int8.top1=91.6 · seed=1Ran with the spec: 5 of its settings found in results/int8-ptq/seed1/config.yaml, all matching.Produced: 2026-10-20 10:00:00 (file)four-of-five criterion undetermined · point only: int8.top1: 91.6 % · mean, 1 seed (rb)best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 1 seed (rb)relative-drop criterion not met · point only: change.top1: -0.1199 % · relative change of the means, paired, 1 seed (rb)rb put the code in git at refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020 removes it.Since this session last looked:- four-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1 (rb decide four-of-five accept|reject|investigate --why "...")- best-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1 (rb decide best-of-five accept|reject|investigate --why "...")- relative-drop misses its rule and no call on it is recorded: INT8 loses at most 0.1% of FP32's top-1 (rb decide relative-drop accept|reject|investigate --why "...")- 4 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed2/, results/int8-ptq/seed3/, results/int8-ptq/seed4/ and 1 more (rb reconcile)Next: rb show four-of-five · rb show best-of-five · rb show relative-drop$ rb run add int8-seeds --from results/int8-ptq/seed2/metrics.json --set seed=2 --config results/int8-ptq/seed2/config.yamlrun5zw7oj on int8-seeds: fp32.top1=91.66, int8.top1=91.58 · seed=2Ran with the spec: 5 of its settings found in results/int8-ptq/seed2/config.yaml, all matching.Produced: 2026-10-20 10:00:00 (file)four-of-five criterion undetermined · point only: int8.top1: 91.59 % · mean, 2 seeds (rb)best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 2 seeds (rb)relative-drop criterion not met · point only: change.top1: -0.1036 % · relative change of the means, paired, 2 seeds (rb)Since this session last looked:- 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed3/, results/int8-ptq/seed4/, results/int8-ptq/seed5/ (rb reconcile)Next: rb show four-of-five · rb show best-of-five · rb show relative-drop$ rb run add int8-seeds --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yamlrunongfds on int8-seeds: fp32.top1=91.74, int8.top1=91.69 · seed=3Ran with the spec: 5 of its settings found in results/int8-ptq/seed3/config.yaml, all matching.Produced: 2026-10-20 10:00:00 (file)four-of-five criterion undetermined · measured: int8.top1: 91.62 [91.48, 91.77] % · mean, t interval, 3 seeds, 95% (rb)best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 3 seeds (rb)relative-drop estimate meets criterion · loss: change.top1: -0.08724 [-0.1685, -0.005936] % · relative change of the means, paired, delta-method t interval, 3 seeds, 95% (rb)Since this session last looked:- 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed4/, results/int8-ptq/seed5/ (rb reconcile)Next: rb show four-of-five · rb show best-of-five · rb show relative-drop$ rb run add int8-seeds --from results/int8-ptq/seed4/metrics.json --set seed=4 --config results/int8-ptq/seed4/config.yamlrunmyetzp on int8-seeds: fp32.top1=91.69, int8.top1=91.57 · seed=4Ran with the spec: 5 of its settings found in results/int8-ptq/seed4/config.yaml, all matching.Produced: 2026-10-20 10:00:00 (file)four-of-five criterion undetermined · measured: int8.top1: 91.61 [91.52, 91.7] % · mean, t interval, 4 seeds, 95% (rb)best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 4 seeds (rb)relative-drop estimate meets criterion · loss: change.top1: -0.09815 [-0.153, -0.04325] % · relative change of the means, paired, delta-method t interval, 4 seeds, 95% (rb)Since this session last looked:- results/int8-ptq/seed5/ holds results written before the record began, which no recorded experiment reads yet (rb reconcile)Next: rb show four-of-five · rb show best-of-five · rb show relative-drop$ rb run add int8-seeds --from results/int8-ptq/seed5/metrics.json --set seed=5 --config results/int8-ptq/seed5/config.yamlrunovg3mt on int8-seeds: fp32.top1=91.72, int8.top1=91.66 · seed=5Ran with the spec: 5 of its settings found in results/int8-ptq/seed5/config.yaml, all matching.Produced: 2026-10-20 10:00:00 (file)four-of-five recorded values meet criterion · measured: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)best-of-five recorded values meet criterion · point only: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)relative-drop estimate meets criterion · loss: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)Next: rb show four-of-five · rb show best-of-five · rb show relative-drop -
Check the file format
These criteria are saved in format 2. rb 0.7.0 and 0.7.1 write nothing to a record with a format-2 file, so upgrade everyone who shares it together.
Terminal window $ head -3 .rb/experiments/int8-seeds/criteria/four-of-five.yamlrb: 2kind: criterionid: four-of-five -
Read each criterion
Terminal window $ rb show four-of-fiveCriterion four-of-five on int8-seeds (primary, own): INT8 reaches 91.58% top-1 on at least 4 of 5 seedsfour-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seedsrecorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58reads: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb)result: measured: int8 on its own, with its interval, so neither better nor worserule: recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4timing: declared after the evidence existed: all 5 units were produced before it was declaredunits: 5 seedsintegrity: okrunbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5Terminal window $ rb show best-of-fiveCriterion best-of-five on int8-seeds (primary, own): INT8's best of 5 seeds reaches 91.65% top-1best-of-five INT8's best of 5 seeds reaches 91.65% top-1recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65reads: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb)result: point only: a best of N is one unit's own value, with no intervalrule: recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65timing: declared after the evidence existed: all 5 units were produced before it was declaredunits: 5 seedsintegrity: okrunbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5Terminal window $ rb show relative-dropCriterion relative-drop on int8-seeds (primary, own): INT8 loses at most 0.1% of FP32's top-1relative-drop INT8 loses at most 0.1% of FP32's top-1estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1%reads: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb)result: loss: the interval clear of 0 on the worse side (higher is better)rule: estimate meets criterion: estimate -0.0916% ≥ -0.1%timing: declared after the evidence existed: all 5 units were produced before it was declaredunits: 5 seedsintegrity: ok· the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole intervalrunbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1runbvd5mg: fp32.top1 91.71 (file) · unit seed=1 · produced before the declaration · seed=1run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2run5zw7oj: fp32.top1 91.66 (file) · unit seed=2 · produced before the declaration · seed=2runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3runongfds: fp32.top1 91.74 (file) · unit seed=3 · produced before the declaration · seed=3runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4runmyetzp: fp32.top1 91.69 (file) · unit seed=4 · produced before the declaration · seed=4runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5runovg3mt: fp32.top1 91.72 (file) · unit seed=5 · produced before the declaration · seed=5
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0