Skip to content

Check a summary's claims against the runs

Check an agent’s summary before you repeat it to your team. Each claim becomes a criterion, a rule rb checks against the runs.

Paste this into your agent

Check every claim in SUMMARY.md against the runs, with rb doing the reading.

My last session wrote SUMMARY.md. Check each claim in it against the runs with Rabbit Brain, and take nothing on the summary's word.

- Run `rb context --model <your model id>` first.
- Record the experiment, and every run from its own file in `results/`.
- Turn each claim into the criterion it states, cited to its line: `--source SUMMARY.md --quote "<the claim's words>"`. Use the claim's own number, with no threshold of yours.
- A claim no criterion can read goes in a note that names the files you read.
- Compute nothing yourself: rb computes the findings.

Finish with `rb status` exactly as printed, and for each claim, what rb says of it.

We ran it on an INT8 ResNet-20 repository with three seeds and this summary:

SUMMARY.md
# Session summary
- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.
- INT8 averages 91.62 top-1 over the three seeds.
- 512 calibration images are enough.
Terminal window
$ rb status
INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10
Kept with rb since 2026-11-10. All 3 runs were recorded afterwards, not as they happened.
15 record files not committed
Does INT8 match FP32 top-1 on CIFAR-10? (open)
On every int8-ptq criterion below: question link declared after the evidence existed: all 3 units were produced before it was declared; units: 3 seeds.
The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here. top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2); rule and question link declared after the evidence existed: all 3 units were produced before it was declared.
The summary's claim that INT8 averages 91.62 top-1 is checked here. int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; reproduced on the estimate: estimate 91.6233 within 91.62 ± 0.005; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [91.48, 91.77] crosses 91.615 and 91.625: reproduced on the estimate, not across the whole interval.
experiment context: int8-ptq · INT8 post-training quantization
note nsctsjr: No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/. · not confirmed
Rules: 1 reproduced, on the estimate alone · 1 not reproduced
Needs judgment (1)
- every-seed-claim is not reproduced and no call on it is recorded: The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
rb decide every-seed-claim accept|reject|investigate --why "..."
The design of 1 experiment is not confirmed: rb confirm lists it.
Facts that ask for nothing (1)
- int8-ptq: 2 criteria were declared after their evidence existed
rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.
rb checked the record with git at 2026-11-10 15:32 UTC: all 15 record files are new, changed or deleted since commit 6eed520f0d01, which does not hold the record.
Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/int8-ptq/criteria/every-seed-claim.yaml (new) and 12 more
To check it yourself, in your terminal: git -C ~/int8-summary status --short --untracked-files=all -- .rb
Rules
every-seed-claim The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
not reproduced · gain · each unit's top1: fp32 vs int8 ≤ 0.05
reads: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb)
result: gain: the interval clear of 0 on the better side (higher is better)
rule: not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2)
timing: declared after the evidence existed: all 3 units were produced before it was declared
units: 3 seeds
int8-mean-claim The summary's claim that INT8 averages 91.62 top-1 is checked here
reproduced on the estimate · measured · int8.top1 = 91.62 ± 0.005
reads: int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb)
result: measured: int8 on its own, with its interval, so neither better nor worse
rule: reproduced on the estimate: estimate 91.6233 within 91.62 ± 0.005
timing: declared after the evidence existed: all 3 units were produced before it was declared
units: 3 seeds
· the interval [91.48, 91.77] crosses 91.615 and 91.625: reproduced on the estimate, not across the whole interval
Experiments
int8-ptq INT8 post-training quantization · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 3
to run again: code unknown · environment unknown · files ok
Runs: 3 recorded on 1 experiment · every receipt is what rb wrote
Next: rb confirm · rb decide every-seed-claim accept|reject|investigate --why "..."
  • Seeds 1 and 2 break the “never loses more than 0.05” claim. The criterion compares FP32 against INT8, so “gain” means FP32 is ahead.
  • The 91.62 average is reproduced on the estimate, not across the whole interval.
  • No run compares calibration sizes, so the 512-images claim gets a note, not a criterion.

The 0.05 claim waits under Needs judgment. The call on it, a decision you record with rb decide, is yours, not the agent’s.

  1. Record the experiment and its settings

    Terminal window
    $ rb experiment add "INT8 post-training quantization" --id int8-ptq -q q-int8 --baseline fp32 --candidate int8 --varies precision,calib_images
    Experiment int8-ptq: INT8 post-training quantization · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · under q-int8
    Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
    Made 2 variants: fp32 (baseline), int8 (candidate); a setting of one is rb spec set int8-ptq fp32.<setting> <value>.
    int8-ptq declares it varies precision, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq <variant>.precision <value> --source <file#key>
    int8-ptq declares it varies calib_images, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq <variant>.calib_images <value> --source <file#key>
    Next: rb spec set int8-ptq --from <config.yaml> --keys "<the keys that define it>" · rb criterion add "<what it should show>" -e int8-ptq -q q-int8 --metric <name> --at-most <x>
    Terminal window
    $ rb spec set int8-ptq fp32.precision fp32 --source "configs/fp32.yaml#precision"
    int8-ptq/fp32.precision = fp32 · verified in configs/fp32.yaml#precision
    $ rb spec set int8-ptq int8.precision int8 --source "configs/int8.yaml#precision"
    int8-ptq/int8.precision = int8 · verified in configs/int8.yaml#precision
    $ rb spec set int8-ptq int8.calib_images 512 --source "configs/int8.yaml#calib_images"
    int8-ptq/int8.calib_images = 512 · verified in configs/int8.yaml#calib_images
    $ rb spec set int8-ptq seed --per-run
    int8-ptq/seed · per-run: each run gives its own, with --set seed=<value> on rb run add
  2. Turn each claim into the criterion it states

    With --source, rb checks that the quoted line is there and holds the criterion’s number. That is why the 0.05 claim is written as FP32 against INT8.

    Terminal window
    $ rb criterion add "The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here" -e int8-ptq -q q-int8 --metric top1 --subject fp32 --reference int8 --at-most 0.05 --each --source SUMMARY.md --quote "it never loses more than 0.05 points of top-1 on any seed" --id every-seed-claim
    every-seed-claim: The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
    each unit's top1: fp32 vs int8 ≤ 0.05 · primary · on int8-ptq
    cited: SUMMARY.md "it never loses more than 0.05 points of top-1 on any seed" · states 0.05 (read '- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.')
    declared; no evidence recorded yet
    Next: rb run add int8-ptq --from <the results file a run wrote> --set seed=<value>
    $ rb criterion add "The summary's claim that INT8 averages 91.62 top-1 is checked here" -e int8-ptq -q q-int8 --metric int8.top1 --equals 91.62 --tolerance 0.005 --source SUMMARY.md --quote "INT8 averages 91.62 top-1 over the three seeds" --id int8-mean-claim
    int8-mean-claim: The summary's claim that INT8 averages 91.62 top-1 is checked here
    int8.top1 = 91.62 ± 0.005 · primary · on int8-ptq
    cited: SUMMARY.md "INT8 averages 91.62 top-1 over the three seeds" · states 91.62 (read '- INT8 averages 91.62 top-1 over the three seeds.')
    declared; no evidence recorded yet
    Next: rb run add int8-ptq --from <the results file a run wrote> --set seed=<value>
  3. Record each run from its own file

    After the third seed:

    Terminal window
    $ rb run add int8-ptq --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml
    runpcz6en on int8-ptq: fp32.top1=91.74, int8.top1=91.69 · seed=3
    Ran with the spec: 1 of its settings found in results/int8-ptq/seed3/config.yaml, matching.
    Produced: 2026-10-20 10:00:00 (file)
    every-seed-claim not reproduced · gain: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb)
    int8-mean-claim reproduced on the estimate · measured: int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb)
    Next: rb show every-seed-claim · rb show int8-mean-claim
  4. Note the claim no criterion can read

    Terminal window
    $ rb note add "No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/." -e int8-ptq
    Note nsctsjr on int8-ptq: No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/.
    Kept verbatim with who wrote it; a note changes no finding.
  5. Read each claim’s criterion

    Terminal window
    $ rb show every-seed-claim
    Criterion every-seed-claim on int8-ptq (primary, cited): The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
    cited: SUMMARY.md "it never loses more than 0.05 points of top-1 on any seed" · states 0.05, read '- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.'
    every-seed-claim The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
    not reproduced · gain · each unit's top1: fp32 vs int8 ≤ 0.05
    reads: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb)
    result: gain: the interval clear of 0 on the better side (higher is better)
    rule: not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2)
    timing: declared after the evidence existed: all 3 units were produced before it was declared
    units: 3 seeds
    integrity: ok
    runikex4f: difference +0.11 (file) · unit seed=1 · produced before the declaration · seed=1
    rungwaqdb: difference +0.08 (file) · unit seed=2 · produced before the declaration · seed=2
    runpcz6en: difference +0.05 (file) · unit seed=3 · produced before the declaration · seed=3

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0