# Rabbit Brain docs, every page # Rabbit Brain docs https://rabbitbrain.ai/docs/ `git status` shows which files you changed. `rb status` shows which results met their pass mark. ## Quick start: a sample study Try Rabbit Brain on a sample study, so you don't need to set up any experiments or use your own research. Just open Claude Code or Codex in an empty folder and paste the prompt below. **Sample study** ```text Clone https://github.com/rabbit-brain/think-step-by-step and work inside it. Install rabbit-brain from PyPI into a virtual environment in that folder. It's open source: https://github.com/rabbit-brain/rb Start rb login in the background and show me the sign-in link it prints. I'll sign in. When I'm signed in, run rb sync and give me the link it prints. Run rb context and tell me in three lines what the study found and what's still open. Don't run any experiments. ``` Your agent gives you a sign-in link (an email code, no password). Then the study opens in your private account on rabbitbrain.ai. No account? Ask your agent to run rb page instead. ## What rb shows you In the warmup cookbook, the plan says warmup must lower validation loss by at least 0.02 nats. Over 3 seeds it lowered it by 0.028, so the rule is met. The rule was written before the runs, and rb says so. ```rb-session frame="terminal" $ rb show warmup-helps Criterion warmup-helps on warmup (primary, own): Warmup lowers final validation loss by at least 0.02 nats warmup-helps Warmup lowers final validation loss by at least 0.02 nats criterion met · gain · change.val_loss ≤ -0.02 (n ≥ 3) reads: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (lower is better) rule: criterion met: estimate -0.02767 ≤ -0.02 timing: declared before the evidence (by production times) units: 3 seeds integrity: ok runttgab7: difference -0.031 (file) · unit seed=1 · seed=1 runvw3726: difference -0.026 (file) · unit seed=2 · seed=2 run2ugudg: difference -0.026 (file) · unit seed=3 · seed=3 ``` ## The loop 1. Your agent records the work Run `rb init ""` in your repository, then paste this into your agent: ```text Use Rabbit Brain to track this: read rb context, then record where the work stands. ``` 2. rb checks and works out the results When the agent records 500 for `warmup_steps`, rb checks that `configs/warmup.yaml` says 500 too. Each result is worked out from the runs whenever you run `rb status` or `rb show`. 3. You decide You accept or reject a result with `rb decide` and your reason. A decision your agent records waits until you run `rb confirm` in your own terminal. ## Try it in a minute `rb tour` builds a small example project in `./rb-tour` and shows each command with rb's real output. With [uv](https://docs.astral.sh/uv/), nothing gets installed: ```sh uvx --from rabbit-brain rb tour ``` ```rb-session frame="terminal" $ rb tour … That is the loop, and rb-tour/.rb/ is an ordinary record, YAML files you can read: poke at it. One criterion meets its rule: three seeds, declared and frozen before the runs, every setting with its source. The other is undetermined: two seeds report step_ms and it asks for three. rb will not round that up, and neither should anyone. Keep going, in the record: cd rb-tour rb ui your screen in this terminal rb open the same, in your browser rb page the record as one HTML file rb export the page to publish, the record if asked rb docs the whole manual rb skill add teach your coding agent the discipline … ``` ## Where to go next - [Install](https://rabbitbrain.ai/docs/install/): Install rb, or let your agent do it, then take the one-minute tour. - [Your first record](https://rabbitbrain.ai/docs/first-record/): Run `rb init` in your repository and paste the line it prints into your agent. - [The record](https://rabbitbrain.ai/docs/concepts/record/): One YAML file per run or rule, in `.rb/`. Each setting keeps the file it came from. - [Cookbooks](https://rabbitbrain.ai/docs/cookbooks/): Eleven tasks with prompts to paste, such as checking an agent's summary against its runs. - [The CLI](https://rabbitbrain.ai/docs/reference/cli/): Every command's `--help`, exactly as rb prints it. - [For agents](https://rabbitbrain.ai/docs/agents/llms-txt/): An agent reads the manual with `rb docs`. It must never edit `.rb/` by hand. --- # Install https://rabbitbrain.ai/docs/install/ rb needs Python 3.10 or later. The package is `rabbit-brain` and the command is `rb`. Wheels cover Linux (x86_64, aarch64), macOS (Apple Silicon, Intel) and Windows (x86_64). On Windows, rb 0.8.4 is a preview. See [issue #2](https://github.com/rabbit-brain/rb/issues/2) for known issues. **pip** In a project's environment, where you can also `import rabbit_brain`: ```sh pip install rabbit-brain ``` **uv** As a tool, in its own environment: ```sh uv tool install rabbit-brain ``` **uvx (no install)** Without installing: ```sh uvx --from rabbit-brain rb tour ``` `rb` is then not on your path. Run later commands as `uvx --from rabbit-brain rb `. Check it: ```rb-session frame="terminal" $ rb --version rabbit-brain 0.8.4 ``` ## Take the tour `rb tour` builds a small project in `./rb-tour` and takes it from a question to three runs. It prints every command with its real output: ```rb-session frame="terminal" $ rb tour $ rb init 'Dropout on a tiny MLP' Project 'dropout-on-a-tiny-mlp' started: ~/try-rb/rb-tour/.rb Rabbit Brain YAML files beside the code: commit .rb/ with it, so the record travels with the work. For the agents here: No agent's instructions here yet. rb skill add puts rb's for one: --claude (Claude Code), --agents-md (Codex and others), --cursor Then paste this line into your agent's chat: Use Rabbit Brain to track this: read rb context, then record where the work stands. $ rb question add 'Can we use dropout 0.1 within our loss and step-time budgets?' --id q1 q1: Can we use dropout 0.1 within our loss and step-time budgets? Next: rb experiment add "" -q q1 --hypothesis "" $ rb experiment add 'dropout 0.1 vs none' --id drop -q q1 --hypothesis 'Dropout 0.1 lowers val_loss' -m 'less co-adaptation' --baseline none --candidate p01 --varies dropout --note 'assumes the validation split stayed fixed' Experiment drop: dropout 0.1 vs none · none (baseline) vs p01 (candidate) · varies dropout · under q1 hypothesis: Dropout 0.1 lowers val_loss (why: less co-adaptation) Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 2 variants: none (baseline), p01 (candidate); a setting of one is rb spec set drop none. . drop declares it varies dropout, but no variant has a value for it yet; give each variant its own: rb spec set drop .dropout --source … (137 more lines) ``` Then `cd rb-tour` and run `rb ui` (terminal) or `rb open` (browser) to look around. ## Updates Installed as a tool (`uv tool install` or `pipx install`), rb upgrades itself through that installer, at most once a day. Installed with pip, rb never touches the environment. When a newer release is out, it prints a one-line notice instead, at most once a day. Neither runs in CI, in an agent session, with `DO_NOT_TRACK` set, with `RB_AUTO_UPDATE=0`, or after `rb config auto-update off`. The update check carries nothing about you. ### A team on two versions rb 0.8.4 saves some rules in format 2, which rb 0.7.0 and 0.7.1 cannot read. Upgrade everyone who shares a record together. [The record's format](https://rabbitbrain.ai/docs/reference/schema/#format-1-and-format-2) has the details. ## What rb sends, and when rb needs the network only for: - `rb clone`, which fetches a bundle and the code it names - `rb login`, `rb sync` and `rb logout`, which connect to rabbitbrain.ai or a hub you name - the daily update check - usage analytics, once you run `rb telemetry enable` in your own terminal Analytics never carry research content, paths, command arguments or names. `rb telemetry preview` shows what would be sent. `DO_NOT_TRACK=1` or `RB_TELEMETRY=0` keeps them off. ## Uninstall `rb remove` takes out only rb's parts: the record, its agent instructions and its hook handlers. Your research files and git history stay. `rb remove --dry-run` shows what it would do first. Then run `pip uninstall rabbit-brain` or `uv tool uninstall rabbit-brain`. - [Next: your first record](https://rabbitbrain.ai/docs/first-record/): Start the record, and give your agent the line to paste. --- # Your first record https://rabbitbrain.ai/docs/first-record/ ## Start the record In your own terminal, in your repository: ```sh rb init "" ``` We ran it in a repository that trains ViT-Tiny with and without label smoothing: ```rb-session frame="terminal" $ rb init "Label smoothing on ViT-Tiny" Project 'label-smoothing-on-vit-tiny' started: ~/vit-smoothing/.rb Rabbit Brain YAML files beside the code: commit .rb/ with it, so the record travels with the work. For the agents here: Claude Code (CLAUDE.md): the skill in .claude/skills/rabbit-brain/SKILL.md; rb's MCP server in a new .mcp.json; rb's session hook in a new .claude/settings.json (SessionStart and Stop run rb journal scan, which looks at the repository and says nothing) Then paste this line into your agent's chat: Use Rabbit Brain to track this: read rb context, then record where the work stands. ``` Then start a new agent session, so it loads rb's skill. ## Hand it to your agent Paste the line `rb init` printed into your agent's chat, and add what you want done: **Track this with Rabbit Brain, and do one piece of work.** Works in any agent that runs terminal commands. ```text Use Rabbit Brain to track this: read rb context, then record where the work stands. Then find out whether label smoothing is worth keeping on our ViT-Tiny, with seeds 1 to 3. ``` ## What the agent did The agent read `rb context` first: ```rb-session frame="terminal" $ rb context # The record: Label smoothing on ViT-Tiny ## First, record where the work stands Nothing is recorded yet, and this project holds work already done: 1 commit. Record where the work stands now, in this turn, from the files, and ask the researcher nothing first. How: rb docs --start. Generated by `rb context` from `.rb/`; read the state from here or `rb status --json`, never from memory or a summary. … (53 more lines) ``` It recorded the question, then the experiment and its settings: ```rb-session frame="terminal" $ rb question add "Is label smoothing worth keeping on our ViT-Tiny?" --id q-smooth --words "find out whether label smoothing is worth keeping on our ViT-Tiny" q-smooth: Is label smoothing worth keeping on our ViT-Tiny? Next: rb experiment add "" -q q-smooth --hypothesis "" ``` ```rb-session frame="terminal" $ rb spec set smooth --from configs/train.yaml --keys "optim.*,data.*" smooth: 3 settings from configs/train.yaml @234b05955a46 · 3 verified ``` Before any run, it recorded the criterion from `docs/plan.md`. A criterion is the rule a result must meet: ```rb-session frame="terminal" $ rb criterion add "Label smoothing 0.1 raises top-1 by at least 0.3 points" -e smooth -q q-smooth --metric change.top1 --at-least 0.3 --min-n 3 --why docs/plan.md:3 --id smoothing-helps smoothing-helps: Label smoothing 0.1 raises top-1 by at least 0.3 points change.top1 ≥ 0.3 (n ≥ 3) · primary · on smooth declared; no evidence recorded yet Next: rb run add smooth --from --set seed= ``` It ran each seed through `rb journal run` and recorded the run with its receipt: what ran and on which commit. After the third seed, the criterion has an interval: ```rb-session frame="terminal" $ rb journal run -- python train.py --seed 3 {"metrics": {"plain.top1": 71.55, "smooth.top1": 72.1}} exit 0. The command, its exit code and what it wrote are in the journal. It printed its metrics: plain.top1=71.55, smooth.top1=72.1. rb reconcile proposes the run of smooth that records them. Since this session last looked: - out/seed3/ holds results no recorded experiment reads yet (rb reconcile) - out/seed3/metrics.json: results landed and are not in the record (rb reconcile) Next: rb reconcile ``` ```rb-session frame="terminal" $ rb run add smooth --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3" --commit 234b05955a4653f6e045af115202ce98dbdb730b runghitef on smooth: plain.top1=71.55, smooth.top1=72.1 · seed=3 Ran with the spec: 4 of its settings found in out/seed3/config.yaml, all matching. Only rb's own files are not committed (.claude/settings.json, .claude/skills/rabbit-brain/SKILL.md, .mcp.json): they are not the run's code, so the receipt finds no changes outside the record. Produced: 2026-11-10 15:32:05 (file) smoothing-helps criterion met · gain: change.top1: +0.54 [+0.4017, +0.6783] % · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show smoothing-helps ``` ## What you'll see Run `rb status` in your own terminal: ```rb-session frame="terminal" $ rb status Label smoothing on ViT-Tiny · label-smoothing-on-vit-tiny 13 record files not committed Is label smoothing worth keeping on our ViT-Tiny? (open) Label smoothing 0.1 raises top-1 by at least 0.3 points. change.top1: +0.54 [+0.4017, +0.6783] % · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); criterion met: estimate +0.54 ≥ 0.3; rule and question link declared before the evidence (by production times). units: 3 seeds. Rules: 1 met The design of 1 experiment is not confirmed: rb confirm lists it. rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 13 record files are new, changed or deleted since commit 234b05955a46, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/smooth/criteria/smoothing-helps.yaml (new) and 10 more To check it yourself, in your terminal: git -C ~/vit-smoothing status --short --untracked-files=all -- .rb Rules smoothing-helps Label smoothing 0.1 raises top-1 by at least 0.3 points criterion met · gain · change.top1 ≥ 0.3 (n ≥ 3) reads: change.top1: +0.54 [+0.4017, +0.6783] % · paired difference, t interval, 3 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (higher is better) rule: criterion met: estimate +0.54 ≥ 0.3 timing: declared before the evidence (by production times) units: 3 seeds Experiments smooth Label smoothing 0.1 vs none · open · plain (baseline) vs smooth (candidate) · varies label_smoothing · not frozen · settings 5 verified · 0 provisional · 0 unknown · 1 per-run · runs 3 to run again: code ok · environment ok · files ok Runs: 3 recorded on 1 experiment · every receipt is what rb wrote Nothing open. Next: rb confirm ``` The Rules block shows the criterion's finding: what rb works out from the runs. [Findings](https://rabbitbrain.ai/docs/concepts/findings/) explains each line. ## Confirm the design The agent wrote the experiment's design, so it reads "not confirmed" until you sign off on it with `rb confirm`: ```rb-session frame="terminal" $ rb confirm Designs waiting for a confirmation (1): smooth: Label smoothing 0.1 vs none questions: q-smooth hypothesis or purpose: not recorded why: not recorded variants: plain (baseline), smooth (candidate) · varies label_smoothing setting optim.lr: 0.001 from configs/train.yaml#optim.lr setting optim.weight_decay: 0.05 from configs/train.yaml#optim.weight_decay setting data.batch_size: 128 from configs/train.yaml#data.batch_size setting seed: per run setting plain.label_smoothing: 0.0 from configs/plain.yaml#label_smoothing setting smooth.label_smoothing: 0.1 from configs/smooth.yaml#label_smoothing criterion smoothing-helps (primary): change.top1 ≥ 0.3 (n ≥ 3) · for q-smooth confirm: rb confirm smooth@f6b6812a74bdbdfd0c6b1dbf8c1ebd0de7390051d0b73fbfb11624b7e88dc0d3 ``` ```rb-session frame="terminal" $ rb confirm smooth@f6b6812a74bdbdfd0c6b1dbf8c1ebd0de7390051d0b73fbfb11624b7e88dc0d3 smooth's design is confirmed. Next: rb status ``` `rb confirm` runs only in your own terminal. An agent session that runs it is refused. - [Connect your agent](https://rabbitbrain.ai/docs/connect/): What rb installs for Claude Code, Codex and Cursor. - [Cookbooks](https://rabbitbrain.ai/docs/cookbooks/): Eleven tasks, each with its prompt and rb's real output. --- # Connect your agent https://rabbitbrain.ai/docs/connect/ `rb init` installs rb's instructions for each agent it finds in the repository. It asks before it changes a file you already have, and `--yes` answers up front. Where it cannot ask, it leaves the file alone and names the command to run. `rb init --dry-run` lists what it would write. [rb with Claude Code, Codex and Cursor](https://rabbitbrain.ai/docs/cookbooks/three-agents/) shows both with real output. **Claude Code** When it finds `.claude/`, `CLAUDE.md` or a Claude Code session, `rb init` installs: - the skill, in `.claude/skills/rabbit-brain/SKILL.md` - rb's MCP server, in `.mcp.json` - rb's [session hook](#the-session-hook) To add them later: ```sh rb skill add --claude --project claude mcp add --scope project rabbit-brain -- rb mcp ``` Then start a new session. **Codex** Codex reads `AGENTS.md`. When the file exists or Codex runs `rb init`, rb adds its section at the end, between two marker lines. Any other agent that reads `AGENTS.md` gets it too. To add it later: ```sh rb skill add --agents-md ``` When `.codex/` is there or Codex runs `rb init`, rb also adds its [session hook](#the-session-hook). Codex runs a project's hooks once it is trusted, in `/hooks`. **Cursor** When it finds `.cursor/` or `.cursorrules`, rb adds a rule in `.cursor/rules/rabbit-brain.mdc` and its [session hook](#the-session-hook). To add them later: ```sh rb skill add --cursor ``` **A plain shell** Every write an agent makes is an rb command you can type: ```sh rb init "INT8 ResNet-20 on CIFAR-10" rb question add "Does INT8 lose at most 0.5 points of top-1 against FP32?" --id q-int8 rb experiment add "INT8 post-training quantization" --id int8-ptq -q q-int8 --baseline fp32 --candidate int8 --varies precision rb spec set int8-ptq int8.precision int8 --source configs/int8.yaml#precision rb metric add top1 --unit % --maximize rb criterion add "INT8 loses at most 0.5 points of top-1 against FP32" -e int8-ptq -q q-int8 --metric change.top1 --at-least -0.5 rb journal run -- python eval.py --seed 1 rb run add int8-ptq --from results/int8-ptq/seed1/metrics.json --set seed=1 rb status ``` [The CLI](https://rabbitbrain.ai/docs/reference/cli/) lists every command. ## What every agent gets - `rb context` prints the record as one briefing. The skill tells the agent to run `rb context --model ` first. - `rb mcp` serves the same commands as MCP tools. ## The session hook rb's session hook runs `rb journal scan --hook ` as a session starts and as each turn ends, so rb looks at the repository even when the agent skips rb's instructions. It prints nothing and has a 10-second limit. rb notes the scan in its journal, not in the record. | Agent | Where | | --- | --- | | Claude Code | `SessionStart` and `Stop`, in `.claude/settings.json` | | Codex | `SessionStart` and `Stop`, in `.codex/hooks.json` | | Cursor | `sessionStart` and `stop`, in `.cursor/hooks.json` | rb's handler goes beside your other hooks, which stay as they are. On Windows, Claude Code needs Git Bash to run the hook. Without it the hook fails, but the session goes on. Cursor's hook and Codex's on Windows look for rb on the PATH alone. ## What stays yours rb records who made each write. A call is a decision on a result (accept, reject, investigate) or on a line of work (continue, park, stop, pivot, ship). Calls are yours. A call recorded in an agent session is proposed, not made, until you approve it with `rb confirm` in your own terminal. rb refuses `rb confirm` in an agent session. An experiment design an agent wrote reads "design not confirmed" until you approve it the same way. [Calls](https://rabbitbrain.ai/docs/concepts/calls/) has the details. --- # The record https://rabbitbrain.ai/docs/concepts/record/ The record is the `.rb/` folder at the root of your repository: plain YAML files you commit with your code. ## What it holds Every item is its own file, and every experiment has its own folder: ```text .rb/project.yaml the project: its title, where it was cloned from .rb/notes/.yaml one file per note on the project .rb/questions/.yaml what the work is trying to find out .rb/experiments//experiment.yaml an experiment: its questions, purpose, hypothesis, shared settings, freeze .rb/experiments//variants/.yaml one file per variant: its role and its own settings .rb/experiments//notes/.yaml one file per note on the experiment .rb/experiments//criteria/.yaml one file per criterion .rb/experiments//runs/.yaml one file per run, with its receipt .rb/experiments//analyses/.yaml one file per analysis .rb/metrics/.yaml the metric catalogue .rb/resources/.yaml datasets, checkpoints, code .rb/decisions/.yaml calls, each with who made it, why, and what it was made on .rb/log.jsonl every write: who made it, in which chat session, how, and the hash of what was written .rb/cache/ local copies, kept out of git ``` Two branches that add runs write different files, so they merge without a conflict. A run's receipt names its command and commit. A criterion is a rule a result must meet ([Criteria](https://rabbitbrain.ai/docs/concepts/criteria/)), and its finding is what rb works out from the runs. A call is a decision on the work, such as accepting a result ([Calls](https://rabbitbrain.ai/docs/concepts/calls/)). ## Where a value came from `rb spec set` records a setting with the file that states it, such as a config key (`configs/train.yaml#optim.lr`) or a table row (`results/sweep.csv#[precision=int8].calib`). rb reads the source on the spot. The source must name the setting: `resnet50` does not state 50, and `adamw` does not state `adam`. A comment never counts. When the source states the value, rb keeps the text it read, the file's hash and the commit. Other settings are provisional (given but not checked, such as a web address), unknown (nobody has found it) or inherited (from a cloned record). rb checks the sources again on every read. A source that states another value is a conflict, and one that no longer states it is stale. Both are problems on every finding of that experiment. To check what a run used, record it with its resolved config (`rb run add ... --config out/seed1/config.yaml`). A run whose config disagrees with the recorded settings is shown and not counted. ## Who wrote what rb never refuses a write because of who makes it. Every write records who made it (you, or the agent rb detects), the chat session and model when the agent gives them, and the reason given. This is in `rb log --json`, a command's `--json` output and `.rb/log.jsonl`. Screens such as `rb status` and `rb open` never show it. They do show agent work that waits for your approval with `rb confirm`, such as "design not confirmed". ## Nothing computed is stored rb never stores findings, or the state of experiments and questions. It computes them on every read. If a page you were sent disagrees with `rb status`, `rb status` is right. ## Hand edits Never edit `.rb/` by hand. A run changed outside rb is not counted, and every finding that depends on a changed file lists the problem. Adding a comment or reformatting does not count as an edit. `rb doctor` lists changed files. `rb doctor --restore` puts back what rb last wrote. `rb doctor --adopt --why "..."` keeps the change and records who adopted it. A file that is not a valid record stops `rb status`, `rb show` and every `--fail-on` gate with `E_STATE_CORRUPT`, and rb says where it breaks. ## How git holds the record rb says how git holds the record, in words such as "all record files committed" or "2 record files not committed". `rb status` says it under the title, and the details further down: ```rb-session frame="terminal" $ rb status … rb checked the record with git at 2026-11-10 15:32 UTC: all 13 record files are new, changed or deleted since commit 234b05955a46, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/smooth/criteria/smoothing-helps.yaml (new) and 10 more To check it yourself, in your terminal: git -C ~/vit-smoothing status --short --untracked-files=all -- .rb … ``` ## What rb sees Every `rb status` says what rb looks at: ```rb-session frame="terminal" $ rb status … rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. … ``` rb also looks in any folder a recorded run named. `rb reconcile` lists results nobody recorded, each with the `rb run add` line that records it. rb reads no run from a report in prose, such as `RESULTS.md`. Each row of a CSV, TSV or JSON Lines table is a run. `rb run add --from results.tsv --where =` records one. `--every-row` records every row `--where` picks as its own run, and reads a SQLite table the same way (`--from results.db --table evals`). [Record one table of many runs](https://rabbitbrain.ai/docs/cookbooks/one-table/) and [Record a folder per run](https://rabbitbrain.ai/docs/cookbooks/run-folders/) show both layouts. ## The format Each file opens with its format, `rb: 1` or `rb: 2`. Only a criterion whose rule is a count, a best of N, a relative change or a cited difference of means is written in format 2 ([Criteria](https://rabbitbrain.ai/docs/concepts/criteria/#four-more-kinds-of-rule)). Both formats are one JSON Schema, [rb-v2.schema.json](https://rabbitbrain.ai/schema/rb-v2.schema.json). `rb schema all` prints it, and [The record's format](https://rabbitbrain.ai/docs/reference/schema/) lists every kind. --- # Criteria https://rabbitbrain.ai/docs/concepts/criteria/ A criterion is the rule a result must meet. Each one covers a single comparison, such as INT8 against FP32. rb records who declared it and when, and checks it against the runs each time the record is read. The outcome is the criterion's finding. ```sh rb criterion add "INT8 loses at most 0.5 points of top-1 against FP32" \ -e int8-ptq -q q-int8 --metric change.top1 --at-least -0.5 --min-n 3 --why "docs/plan.md:3" ``` ## What it compares `--metric` names a number the runs report: | `--metric` | Reads | | --- | --- | | `top1` | the number named `top1`, or the only candidate's when runs report it per variant | | `int8.top1` | the `int8` variant's number | | `candidate.top1` | the candidate's number | | `change.top1` | the candidate's number minus the baseline's, from the same run | | `top1` with `--subject fp32 --reference int8` | FP32's number against INT8's | Record each variant's own numbers, such as `{"fp32": {"top1": 91.71}, "int8": {"top1": 91.6}}`. rb works out `change.top1` itself. ## The rule The estimate is the mean over units. A unit is an independent repeat, such as a seed. | Rule | Holds when | | --- | --- | | `--at-most X`, `--at-least X` | the estimate is at most X, or at least X | | `--equals X --tolerance T` | the estimate is within T of X | | `--each` | every unit meets the threshold | | `--bound` | the interval's bound meets the threshold | | `--better` | the interval is clear of 0 on the better side | | `--noninferior M` | the interval's worse bound is inside M | | `--equivalent M` | the interval is inside ±M | `--better`, `--noninferior` and `--equivalent` need an interval, which rb gives from 3 units. `--better` and `--noninferior` also need to know which way is better, or rb refuses them: `rb metric add top1 --maximize`. `--min-n 3` sets the fewest units the criterion needs. With fewer, it is undetermined. ## Four more kinds of rule rb 0.8.4 has four more kinds of rule: | Rule | Reads | | --- | --- | | `--count K --of N` | whether at least K of the N units meet the threshold. Undetermined unless the runs hold N units. | | `--best --of N` | the best unit's own value, with `--at-most` or `--at-least`. It has no interval. | | `--relative` | the relative change of the arms' means, (subject - reference) / reference, in percent. The threshold is a percent too. | | `--means`, with `--source` and `--quote` | the cited number as a difference of the arms' means over all their units | Only a criterion with one of these rules is saved in format 2 (`rb: 2`). See [The record's format](https://rabbitbrain.ai/docs/reference/schema/). [Read a rule over seeds](https://rabbitbrain.ai/docs/cookbooks/rules-over-seeds/) has worked examples. ## Where the threshold comes from A threshold comes from a file or from you, never from the agent: - A plan: `--why "docs/plan.md:3"`. rb refuses the criterion if the line is missing, is a comment, or does not state the threshold. - A paper or other document: `--source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6"`. rb checks that the file holds the quote and that it states the number. - Your words: `--words "..."`, kept verbatim. Words an agent passes show as `words (as given)`, since rb cannot know you said them. ## Role `--role primary`, the default, is what the work claims. `secondary` and `exploratory` are context. A `gate` only steers where effort goes. A missed gate refutes nothing. It asks you to decide whether to park or continue the line of work. ## When it was declared Every finding says when its criterion was written, compared with its evidence: | rb says | When | | --- | --- | | declared before the evidence | it was written before its runs were produced | | declared after the evidence existed | some or all of its runs already existed. rb says how many. | | declared after the data was seen | the data it evaluates on was looked at first | | order unknown | some evidence has no known time, such as numbers typed in with no `--produced-at` | Evidence that existed first still counts, but rb never reads it as if the criterion came first. [Declare the criterion before the runs](https://rabbitbrain.ai/docs/cookbooks/criterion-before-runs/) shows both timings side by side. ## Freezing `rb freeze -m "before the runs"` locks the experiment's variants, settings and criteria, and records their hash. After that, a change fails with `E_FROZEN` unless the command adds `--amend --why "..."`. rb keeps each amendment with who made it. ## Which question it answers `-q ` links a criterion to a question, and only linked criteria count for that question. `--when-met "..."` keeps your words on what meeting the rule would mean. If the rule misses, rb does not claim the opposite. --- # Findings https://rabbitbrain.ai/docs/concepts/findings/ A finding is rb's reading of a criterion, a rule a result must meet, against the runs. rb computes it on every read and never stores it. `rb show` after five seeds: ```rb-session frame="terminal" $ rb show interval-within-budget Criterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) reads: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb) result: unresolved: the interval includes 0 rule: criterion met: the interval's worse bound +0.01338 inside 0.02 timing: declared before the evidence (by production times) units: 5 seeds integrity: ok rungxyf6p: difference +0.008 (file) · unit seed=1 · seed=1 runbklxmd: difference -0.003 (file) · unit seed=2 · seed=2 runj5w6lb: difference +0.014 (file) · unit seed=3 · seed=3 runrevodj: difference +0.003 (file) · unit seed=4 · seed=4 rungtlduu: difference +0.006 (file) · unit seed=5 · seed=5 ``` ## What each line says | Line | What it says | | --- | --- | | `reads` | The number rb read, its interval and how it was computed. `(rb)` means rb computed it from the runs. | | `result` | What the number shows. `gain` or `loss`: the interval is clear of 0 on the better or worse side. `unresolved`: it includes 0. `equivalent`: inside an `--equivalent` margin. `point only`: no interval yet. `measured`: one variant's number on its own. | | `rule` | Whether the rule is met. `criterion undetermined`: it needs an interval, more units or the metric's direction. `estimate meets criterion`: it holds, but with no interval or one that crosses the threshold. | | `timing` | Whether the criterion was declared before its evidence. See [Criteria](https://rabbitbrain.ai/docs/concepts/criteria/#when-it-was-declared). | | `units` | How many independent units the reading rests on, here seeds. | | `integrity` | Problems with what the finding depends on, or `ok`. | Rule and result are separate: above, the rule is met while the result is unresolved. ## How rb reads the runs - Each run gives one difference between the two arms compared: the subject's number minus the reference's. - Runs with the same per-run values (two runs at `seed=2`) are one unit, their numbers averaged. - The estimate is the mean over units. From 3 units, rb gives a 95% Student t interval. - Arms recorded in separate runs pair up by unit ("paired by seed"). Arms that never share a unit are compared unpaired (Welch). - Declare a setting that only numbers repeats, such as an epoch, with `rb spec set population.repeat epoch --no-verify` (or `seed,epoch`). rb then never offers it as the unit. ## As seeds come in With one or two seeds there is no interval, so `estimate meets criterion` does not mean it works. Two criteria, one on the estimate and one on the interval, after the first seed: ```rb-session frame="terminal" $ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1" rungxyf6p on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1 Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching. Produced: 2026-11-10 15:32:22 (file) mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb) interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb) rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it. Next: rb show mean-within-budget · rb show interval-within-budget ``` After the third, the interval appears but still reaches past 0.02, and the second criterion asks for 5 seeds: ```rb-session frame="terminal" $ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3" runj5w6lb on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3 Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching. Produced: 2026-11-10 15:32:23 (file) mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb) interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show mean-within-budget · rb show interval-within-budget ``` [Get an interval, not a single number](https://rabbitbrain.ai/docs/cookbooks/interval-from-seeds/) walks through all five seeds. ## Integrity The common problems: | Problem | What it means | | --- | --- | | `confound` | the arms differ in a setting the experiment does not say it varies, and nobody accepted it | | `conflict`, `stale` | a setting's source states another value, or no longer states it | | `unknown` | a required setting nobody has found | | `not_comparable` | a setting differs from the value a cited source used, or rb's paired reading cannot give a cited difference of means | | `drift` | a frozen experiment no longer matches its freeze | | `edited_outside_rb` | a record it depends on was changed by hand | A provisional setting (given but not checked), numbers typed in, a run whose config says the code had uncommitted changes, and arms that differ in a per-run setting are notes on a finding, not problems. ## In CI `rb status --fail-on ` exits 1 when the record fails that gate. Most CI jobs want `unestablished`. It fails on a primary criterion that is not met, undetermined, untested or not shown to be declared before what it reads, on any integrity problem, and on a record changed outside rb. ```sh rb status --fail-on unestablished ``` Check out the whole history in CI (`fetch-depth: 0` with `actions/checkout`). rb tells a clean merge from a hand edit only through git's history. `rb docs` says what each gate checks. --- # Calls https://rabbitbrain.ai/docs/concepts/calls/ A call is a decision on the work. rb records it with your reason and the finding it was made on, meaning what rb read from the runs at that time. There are two kinds: - `accept`, `reject` or `investigate` a criterion (a rule a result must meet), an experiment, a question or a setting. - `continue`, `park`, `stop`, `pivot` or `ship` an experiment or a question, to steer the line of work. ```sh rb decide --why "" ``` A finding never makes a call, and a call never changes a finding. An experiment's state, such as parked or shipped, comes from its latest call. ## Calls are yours An agent records a call only when you ask, with your words in `--words`. The call is "proposed, not made" until you approve it with `rb confirm` in your own terminal: ```rb-session frame="terminal" $ rb decide stricter-warmup-bar reject --why "docs/plan.md:3 sets the bar at 0.02 nats" --words "Reject the 0.03 bar. The plan says 0.02." dkk6sym: proposed, not made: reject stricter-warmup-bar · note: docs/plan.md:3 sets the bar at 0.02 nats · words (as given): "Reject the 0.03 bar. The plan says 0.02." It changes nothing until it is confirmed: rb confirm dkk6sym, in your own terminal. Since this session last looked: - stricter-warmup-bar misses its rule and the call recorded on it is not made yet: it waits for a confirmation: Warmup lowers final validation loss by at least 0.03 nats (rb decide stricter-warmup-bar accept|reject|investigate --why "...") - warmup amended (new criterion stricter-warmup-bar: change.val_loss ≤ -0.03 (n ≥ 3)): a stricter bar, set after seeing the runs - criterion stricter-warmup-bar added on warmup (change.val_loss ≤ -0.03 (n ≥ 3)): Warmup lowers final validation loss by at least 0.03 nats Next: rb status ``` `rb status` lists it: ```rb-session frame="terminal" $ rb status … Proposed, not made (1) - reject stricter-warmup-bar · note: docs/plan.md:3 sets the bar at 0.02 nats · words (as given): "Reject the 0.03 bar. The plan says 0.02." · confirm: rb confirm dkk6sym … ``` You confirm it: ```rb-session frame="terminal" $ rb confirm dkk6sym d7tsyb3: rejected stricter-warmup-bar · note: docs/plan.md:3 sets the bar at 0.02 nats · your words: "Reject the 0.03 bar. The plan says 0.02." (confirms dkk6sym) made on stricter-warmup-bar: -0.02767 [-0.03484, -0.0205], n 3 (rb): gain, criterion not met Next: rb status ``` `rb confirm` is refused inside an agent session. On its own, it lists everything waiting, and ends with one command that confirms it all as listed. ## Designs are yours too An experiment's design is how it tests its question: its questions, hypothesis or purpose, variants, settings, criteria, gate and freeze. If an agent wrote any part, the experiment shows "design not confirmed" until you confirm it: ```sh rb confirm # lists the design, every part of it rb confirm @ # confirms it exactly as listed ``` ## Calls made before the record A call made before rb was in the repository is recorded as a past call: ```sh rb decide int8 park --why "..." --past --when "2026-05-07" --source docs/calls.md:3 --quote "We park INT8 for now" ``` rb checks that the quote is in the file, and shows when git says the line was committed and whether that was before the record began. ## Taking something back `rb retract --why "..."` takes back anything, such as a run, a criterion, a note or a call. It stays on the record with who took it back and why, but stops counting. An agent session cannot take back your call, only propose another. --- # Sharing https://rabbitbrain.ai/docs/concepts/sharing/ The record lives in your repository, and leaves your machine only when you send it. rb has four ways to show it: | Command | What it writes | What travels | | --- | --- | --- | | `rb page` | One self-contained, read-only HTML file | The page | | `rb open` | Nothing. It serves the record live in your browser, on 127.0.0.1 only. | Nothing | | `rb export` | The page, a README and a hash manifest, for any static host | The page | | `rb export --with-record` | A bundle: the record, the files its checks read, the page and a manifest of every file's hash | The record too | A small record's page, written with `rb export --with-record --url`: ## What a page leaves out A page names no writer, no words you gave an agent, no decision waiting for your approval, no machine platform, no branch and no path. ## Which commit a page comes from A page's About section says whether the record was committed, and at which commit. `rb page`, `rb export` and `rb sync` print the same line: ```rb-session frame="terminal" $ rb page rb-page.html written: the project as one file, readable anywhere; send it to anyone. It is the record as computed now; regenerate after new runs. If it ever disagrees with rb status, rb status is right. The page says: all record files committed · at commit 2d3d23f523f2. ``` ## What the record carries A bundle made with `--with-record` carries the whole record. rb says what that includes each time: ```rb-session frame="terminal" $ rb export --with-record --zip --out ~/int8-record.zip ~/int8-record.zip written: the whole record, the page, and a manifest hashing every file. Readable without rb; rb status inside record/ reopens it. Regenerate after new runs. The record keeps who made each write (human: or agent:, and the channel it came through), an agent's chat session, name, version, provider and model, the user's words passed with --words, and each run's host, platform and branch. Anyone who can read the bundle can read all of it. The page says: all record files committed · at commit 2d3d23f523f2. Code repository recorded: https://example.org/ada/int8-resnet.git at 2d3d23f523f2. Pass --no-repo to leave it out. Next: rb clone ~/int8-record.zip · rb export --with-record --url (make it publishable) ``` ## Publishing a bundle `rb export --with-record --url
` writes a folder to serve at that address, with the page and the `bundle.zip` others clone: ```rb-session frame="terminal" $ rb export --with-record --url https://example.org/int8-resnet/ --out ~/int8-site ~/int8-site written: the whole record, the page, and a manifest hashing every file. Readable without rb; rb status inside record/ reopens it. Regenerate after new runs. The record keeps who made each write (human: or agent:, and the channel it came through), an agent's chat session, name, version, provider and model, the user's words passed with --words, and each run's host, platform and branch. Anyone who can read the bundle can read all of it. The page says: all record files committed · at commit 2d3d23f523f2. Code repository recorded: https://example.org/ada/int8-resnet.git at 2d3d23f523f2. Pass --no-repo to leave it out. Publish the directory at the address you gave; page.html is the reading surface and anyone reproduces it with: rb clone https://example.org/int8-resnet/bundle.zip ``` The page's "Rerun this research" panel holds a prompt that has a coding agent clone the record, get the code, rerun the runs and report what reproduced. [Share the record, and rerun it](https://rabbitbrain.ai/docs/cookbooks/share-a-record/) shows a rerun from start to end. ## Starting from someone else's record `rb clone ` starts a new line of work from a bundle. If the bundle names its code, the folder is a checkout of that code at the bundle's commit, on a new branch `rerun`. Each criterion (a rule a result must meet) shows untested until your runs test it, then reproduced or not reproduced. The parent's runs and decisions stay in the parent's record. ## A team on two versions rb 0.8.4 saves a criterion in format 2 only when it uses one of the [four more kinds of rule](https://rabbitbrain.ai/docs/concepts/criteria/#four-more-kinds-of-rule). rb 0.8.0 to 0.8.3 read those files as their own. rb 0.7.0 and 0.7.1 write nothing to a record with a format-2 file, refuse to clone a bundle that holds one, and say to upgrade. [The record's format](https://rabbitbrain.ai/docs/reference/schema/#format-1-and-format-2) says what each version does. Upgrade everyone who shares a record together. ## rabbitbrain.ai [rabbitbrain.ai](https://rabbitbrain.ai) serves shared record pages, and sample records at [rabbitbrain.ai/demo](https://rabbitbrain.ai/demo). Make an account at [rabbitbrain.ai/signin](https://rabbitbrain.ai/signin). `rb login` signs a machine in, and `rb sync` keeps a private copy of a record's page in your account. `rb sync --with-record` sends the record too, so readers of a share link can rerun it. Share links are made on the site, never by rb, and can be revoked. Before `rb login`, `rb sync` says what is missing: ```rb-session frame="terminal" $ rb sync rb sync: No credential for rabbitbrain.ai on this machine. → Run rb login once per machine, or set RB_HUB_TOKEN for CI. ``` --- # Cookbooks https://rabbitbrain.ai/docs/cookbooks/ Each cookbook is one task: the prompt to paste into your coding agent, and rb's output from running it. - [Start tracking a project already under way](https://rabbitbrain.ai/docs/cookbooks/project-under-way/): Record where the work stands from the repository's own files, with no questions asked first. - [Record one table of many runs](https://rabbitbrain.ai/docs/cookbooks/one-table/): Each row of results.tsv is a run, recorded with one command per arm. - [Record a folder per run](https://rabbitbrain.ai/docs/cookbooks/run-folders/): Each run's own folder, recorded from what its config says. - [Check a summary's claims against the runs](https://rabbitbrain.ai/docs/cookbooks/check-a-summary/): Turn each claim in an agent's summary into a criterion: a rule rb checks against the runs. - [Declare the criterion before the runs](https://rabbitbrain.ai/docs/cookbooks/criterion-before-runs/): Declare what counts and freeze it, so rb shows it came first. - [Compare against a published baseline](https://rabbitbrain.ai/docs/cookbooks/published-baseline/): Cite the paper's number and settings. rb says reproduced or not reproduced. - [Get an interval, not a single number](https://rabbitbrain.ai/docs/cookbooks/interval-from-seeds/): Run seeds until the criterion has the interval it needs. - [Read a rule over seeds](https://rabbitbrain.ai/docs/cookbooks/rules-over-seeds/): Rules over the seeds, such as a count or a best of N. - [Share the record, and rerun it](https://rabbitbrain.ai/docs/cookbooks/share-a-record/): A page your advisor reads, and a bundle their agent clones and reruns. - [Rerun a run from its receipt](https://rabbitbrain.ai/docs/cookbooks/rerun-from-receipt/): A run's receipt says what ran and on which commit. Rerun it on that commit as a check. - [rb with Claude Code, Codex and Cursor](https://rabbitbrain.ai/docs/cookbooks/three-agents/): One record, whichever agent writes to it. ## About the output on these pages Every output block is real, from rb 0.8.4. Each cookbook ran on a small project whose script writes fixed numbers per seed. Commands marked as yours ran in a plain terminal, the rest in an agent session as Claude Code runs them. After the runs, we replaced the folder's path with `~` and cut long outputs where a line says so. Ids such as `run4k2xa9`, hashes and times differ in your run. --- # Start tracking a project already under way https://rabbitbrain.ai/docs/cookbooks/project-under-way/ You have weeks of work in a repository and no record of it. Start the record from what the files say. ## Paste this into your agent Run `rb init ""` in your own terminal and paste the line it prints: **Record where this repository's work stands now, from its own files.** This is the line rb init prints, word for word. ```text Use Rabbit Brain to track this: read rb context, then record where the work stands. ``` The repository we ran it on holds an INT8 post-training quantization of a ResNet-20 on CIFAR-10: - `configs/` for training, FP32 and INT8 - the plan, `docs/plan.md`: "INT8 ships if it loses at most 0.5 points of top-1 against FP32." - `results/int8-ptq/`: three seeds and a report ## What you'll see Once the agent is done, in your own terminal: ```rb-session frame="terminal" $ rb status INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10 Kept with rb since 2026-11-10. All 3 runs were recorded afterwards, not as they happened. 14 record files not committed Last reconcile, 2026-11-10 15:32 UTC: 1 re-indexed (int8-ptq) · 1 result quoted · 1 next step quoted Does INT8 lose at most 0.5 points of top-1 against FP32 on CIFAR-10? (open) INT8 loses at most 0.5 points of top-1 against FP32. change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb); loss: the interval clear of 0 on the worse side (higher is better); criterion met: estimate -0.08 ≥ -0.5; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. units: 3 seeds. experiment context: int8-ptq · INT8 post-training quantization result: "INT8 top-1 is 91.62 against FP32's 91.70 over three seeds: 0.08 points lower." (README.md:7) next: "Try per-channel calibration with 128 images." (README.md:11) note nrind62: The question is docs/plan.md:3 turned into a question: "INT8 ships if it loses at most 0.5 points of top-1 against FP32." · not confirmed git clean Rules: 1 met The design of 1 experiment is not confirmed: rb confirm lists it. Facts that ask for nothing (1) - int8-ptq: 1 criterion was declared after its evidence existed rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 14 record files are new, changed or deleted since commit 93d9471a93e2, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/int8-ptq/criteria/top1-drop-budget.yaml (new) and 11 more To check it yourself, in your terminal: git -C ~/int8-resnet status --short --untracked-files=all -- .rb Rules top1-drop-budget INT8 loses at most 0.5 points of top-1 against FP32 criterion met · loss · change.top1 ≥ -0.5 reads: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: criterion met: estimate -0.08 ≥ -0.5 timing: declared after the evidence existed: all 3 units were produced before it was declared units: 3 seeds Experiments int8-ptq INT8 post-training quantization · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 7 verified · 0 provisional · 0 unknown · 1 per-run · runs 3 to run again: code unknown · environment unknown · files ok Runs: 3 recorded on 1 experiment · every receipt is what rb wrote Next: rb confirm ``` `top1-drop-budget` is the id of the plan's criterion, a rule the runs are checked against. The experiment shows "code unknown · environment unknown" because the runs were made before rb was watching. The experiment's design stays the agent's until you confirm it as yours: ```rb-session frame="terminal" $ rb confirm Designs waiting for a confirmation (1): int8-ptq: INT8 post-training quantization questions: q-int8 hypothesis or purpose: not recorded why: not recorded variants: fp32 (baseline), int8 (candidate) · varies precision, calib_images setting optim.lr: 0.1 from configs/train.yaml#optim.lr setting optim.momentum: 0.9 from configs/train.yaml#optim.momentum setting data.batch_size: 128 from configs/train.yaml#data.batch_size setting eval.batch_size: 256 from configs/train.yaml#eval.batch_size setting seed: per run setting fp32.precision: "fp32" from configs/fp32.yaml#precision setting int8.precision: "int8" from configs/int8.yaml#precision setting int8.calib_images: 512 from configs/int8.yaml#calib_images criterion top1-drop-budget (primary): change.top1 ≥ -0.5 · for q-int8 confirm: rb confirm int8-ptq@775110fdf0ef6a94a6d1aa30ed23b790454a0151cf4aa4f991f701e30af6773c note nrind62 on int8-ptq stays not confirmed: a note is not part of the design. To say one as yours, write it in your own terminal: rb note add "" -e int8-ptq, then rb retract --why "". ``` ```rb-session frame="terminal" $ rb confirm int8-ptq@775110fdf0ef6a94a6d1aa30ed23b790454a0151cf4aa4f991f701e30af6773c int8-ptq's design is confirmed. note nrind62 on int8-ptq stays not confirmed: a note is not part of the design. To say one as yours, write it in your own terminal: rb note add "" -e int8-ptq, then rb retract --why "". Next: rb status ``` ## What rb did, step by step 1. Read the briefing `rb context` points the agent at `rb docs --start`, the protocol for a project already under way: ```rb-session frame="terminal" $ rb context # The record: INT8 ResNet-20 on CIFAR-10 ## First, record where the work stands Nothing is recorded yet, and this project holds work already done: 2 commits and 1 results folder. Record where the work stands now, in this turn, from the files, and ask the researcher nothing first. How: rb docs --start. … (57 more lines) ``` 2. Read what the repository already says `rb reconcile` quotes the report in each results folder word for word. It writes nothing. ```rb-session frame="terminal" $ rb reconcile The repository at commit 93d9471a93e2: 1 results folder. int8-ptq results/int8-ptq/ new, not in the record title: "INT8 post-training quantization" (README.md:1) says: "Three seeds of FP32 and INT8, evaluated with `python eval.py --seed `." (README.md:3) result: "INT8 top-1 is 91.62 against FP32's 91.70 over three seeds: 0.08 points lower." (README.md:7) next: "Try per-channel calibration with 128 images." (README.md:11) files: 1 report · 6 results files git: all 7 report and results files are tracked · folder unchanged since the last commit rb reconcile --apply keeps this in the record: 1 experiment to add. ``` 3. Record the question and the experiment ```rb-session frame="terminal" $ rb question add "Does INT8 lose at most 0.5 points of top-1 against FP32 on CIFAR-10?" --id q-int8 q-int8: Does INT8 lose at most 0.5 points of top-1 against FP32 on CIFAR-10? Next: rb experiment add "" -q q-int8 --hypothesis "" ``` ```rb-session frame="terminal" $ rb experiment add "INT8 post-training quantization" --id int8-ptq -q q-int8 --baseline fp32 --candidate int8 --varies precision,calib_images Experiment int8-ptq: INT8 post-training quantization · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · under q-int8 Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 2 variants: fp32 (baseline), int8 (candidate); a setting of one is rb spec set int8-ptq fp32. . int8-ptq declares it varies precision, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq .precision --source int8-ptq declares it varies calib_images, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq .calib_images --source Next: rb spec set int8-ptq --from --keys "" · rb criterion add "" -e int8-ptq -q q-int8 --metric --at-most ``` 4. Cite every setting to the file that states it ```rb-session frame="terminal" $ rb spec set int8-ptq --from configs/train.yaml --keys "optim.*,data.*,eval.*" int8-ptq: 4 settings from configs/train.yaml @93d9471a93e2 · 4 verified $ rb spec set int8-ptq fp32.precision fp32 --source "configs/fp32.yaml#precision" int8-ptq/fp32.precision = fp32 · verified in configs/fp32.yaml#precision $ rb spec set int8-ptq int8.precision int8 --source "configs/int8.yaml#precision" int8-ptq/int8.precision = int8 · verified in configs/int8.yaml#precision $ rb spec set int8-ptq int8.calib_images 512 --source "configs/int8.yaml#calib_images" int8-ptq/int8.calib_images = 512 · verified in configs/int8.yaml#calib_images $ rb spec set int8-ptq seed --per-run int8-ptq/seed · per-run: each run gives its own, with --set seed= on rb run add ``` 5. Record the plan's criterion, cited to its line rb checks that `docs/plan.md:3` is there and states the criterion's threshold. ```rb-session frame="terminal" $ rb metric add top1 --unit % --maximize Metric top1 (%): higher is better ``` ```rb-session frame="terminal" $ rb criterion add "INT8 loses at most 0.5 points of top-1 against FP32" -e int8-ptq -q q-int8 --metric change.top1 --at-least -0.5 --why docs/plan.md:3 --id top1-drop-budget top1-drop-budget: INT8 loses at most 0.5 points of top-1 against FP32 change.top1 ≥ -0.5 · primary · on int8-ptq declared; no evidence recorded yet Next: rb run add int8-ptq --from --set seed= ``` 6. Record each run from its own file ```rb-session frame="terminal" $ rb run add int8-ptq --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml runrpxwux on int8-ptq: fp32.top1=91.71, int8.top1=91.6 · seed=1 Ran with the spec: 5 of its settings found in results/int8-ptq/seed1/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) top1-drop-budget estimate meets criterion · point only: change.top1: -0.11 % · paired difference, 1 seed (rb) rb put the code in git at refs/rb/snapshots/49a2f7fb6a172a9a46a4624957355aec6732c343, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/49a2f7fb6a172a9a46a4624957355aec6732c343 removes it. Next: rb show top1-drop-budget $ rb run add int8-ptq --from results/int8-ptq/seed2/metrics.json --set seed=2 --config results/int8-ptq/seed2/config.yaml runhc2lw2 on int8-ptq: fp32.top1=91.66, int8.top1=91.58 · seed=2 Ran with the spec: 5 of its settings found in results/int8-ptq/seed2/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) top1-drop-budget estimate meets criterion · point only: change.top1: -0.095 % · paired difference, 2 seeds (rb) Next: rb show top1-drop-budget $ rb run add int8-ptq --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml run4ssgrk on int8-ptq: fp32.top1=91.74, int8.top1=91.69 · seed=3 Ran with the spec: 5 of its settings found in results/int8-ptq/seed3/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) top1-drop-budget criterion met · loss: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show top1-drop-budget ``` 7. Keep what the report says ```rb-session frame="terminal" $ rb reconcile --apply The repository at commit 93d9471a93e2: 1 results folder. Kept in the record: 1 re-indexed (int8-ptq) · 1 result quoted · 1 next step quoted. int8-ptq results/int8-ptq/ in the record title: "INT8 post-training quantization" (README.md:1) says: "Three seeds of FP32 and INT8, evaluated with `python eval.py --seed `." (README.md:3) result: "INT8 top-1 is 91.62 against FP32's 91.70 over three seeds: 0.08 points lower." (README.md:7) next: "Try per-channel calibration with 128 images." (README.md:11) files: 1 report · 6 results files git: all 7 report and results files are tracked · folder unchanged since the last commit ``` ## Runs kept another way - Each run in its own folder: [Record a folder per run](https://rabbitbrain.ai/docs/cookbooks/run-folders/). - Runs as rows of one table, such as `results.tsv`: [Record one table of many runs](https://rabbitbrain.ai/docs/cookbooks/one-table/). --- # Record one table of many runs https://rabbitbrain.ai/docs/cookbooks/one-table/ Your training script adds a row to `results.tsv` after each run. rb records each row as its own run, and checks the claim in `RESULTS.md` against those runs. ## Paste this into your agent **Record every row of results.tsv as a run, and check what RESULTS.md says.** ```text My runs are the rows of results.tsv. Record them with Rabbit Brain, and check what RESULTS.md says against them. - Run `rb context --model ` first, then `rb reconcile`. - Record the question and the experiment, with each setting cited to the config that states it. - Record every row of results.tsv as its own run, one `rb run add ... --every-row` per arm, with each row's seed and finish time. - Turn the claim in RESULTS.md into the criterion it states, cited to its line. - Compute nothing yourself: rb computes the findings. Finish with `rb reconcile` and `rb status` exactly as printed. ``` The repository trains a ViT-Tiny on CIFAR-100 with no label smoothing and with smoothing 0.1, four seeds each: ```rb-session frame="terminal" $ head -n 4 results.tsv smoothing seed val_acc finished_at 0.0 1 71.42 2026-10-27T09:00:00Z 0.1 1 72.06 2026-10-27T10:00:00Z 0.0 2 71.18 2026-10-27T11:00:00Z ``` ```rb-session frame="terminal" $ cat RESULTS.md # Results Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds. ``` ## What you'll see `rb status` in your own terminal, once the agent is done: ```rb-session frame="terminal" $ rb status Label smoothing on a small ViT · label-smoothing-on-a-small-vit Kept with rb since 2026-11-10. All 8 runs were recorded afterwards, not as they happened. 18 record files not committed Does label smoothing 0.1 raise validation accuracy on ViT-Tiny? (open) Label smoothing 0.1 raises validation accuracy by about 0.66 points. change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005; rule and question link declared after the evidence existed: all 4 units were produced before it was declared. units: 4 seeds from 8 runs; paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit; no run config was compared with the spec (record runs with --config); the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval. Rules: 1 reproduced, on the estimate alone The design of 1 experiment is not confirmed: rb confirm lists it. Facts that ask for nothing (1) - ls: 1 criterion was declared after its evidence existed rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 18 record files are new, changed or deleted since commit d17af2687869, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/ls/criteria/ls-gain-claim.yaml (new) and 15 more To check it yourself, in your terminal: git -C ~/label-smoothing status --short --untracked-files=all -- .rb Rules ls-gain-claim Label smoothing 0.1 raises validation accuracy by about 0.66 points reproduced on the estimate · gain · change.val_acc = 0.66 ± 0.005 reads: change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (higher is better) rule: reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005 timing: declared after the evidence existed: all 4 units were produced before it was declared units: 4 seeds from 8 runs · paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit · no run config was compared with the spec (record runs with --config) · the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval Experiments ls Label smoothing 0.1 vs none · open · none (baseline) vs ls01 (candidate) · varies smoothing · not frozen · settings 5 verified · 0 provisional · 0 unknown · 1 per-run · runs 8 to run again: code unknown · environment unknown · files ok Runs: 8 recorded on 1 experiment · every receipt is what rb wrote Next: rb confirm ``` - A criterion is a rule a result must meet. The claim in `RESULTS.md` is recorded as the criterion `ls-gain-claim`. - `ls-gain-claim` reads **reproduced on the estimate**: +0.66 is within 0.005 of 0.66, but the interval is wider. - "no run config was compared with the spec": the runs came from rows, with no `--config`. ## What rb did, step by step 1. Read the repository ```rb-session frame="terminal" $ rb reconcile The repository at commit d17af2687869: 2 results files in the top folder. Results files in no results folder (2): RESULTS.md: prose: rb reads no run from it; a number there is a cited rule a number it states is checked by a cited rule: rb criterion add "" -e --metric --equals --tolerance --source RESULTS.md --quote "" results.tsv: a TSV table of 8 rows, numbers in smoothing, seed, val_acc a row is a run: rb run add --from results.tsv --where =,... ``` 2. Record the question, experiment and settings ```rb-session frame="terminal" $ rb question add "Does label smoothing 0.1 raise validation accuracy on ViT-Tiny?" --id q-ls q-ls: Does label smoothing 0.1 raise validation accuracy on ViT-Tiny? Next: rb experiment add "" -q q-ls --hypothesis "" ``` ```rb-session frame="terminal" $ rb experiment add "Label smoothing 0.1 vs none" --id ls -q q-ls --baseline none --candidate ls01 --varies smoothing Experiment ls: Label smoothing 0.1 vs none · none (baseline) vs ls01 (candidate) · varies smoothing · under q-ls Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 2 variants: none (baseline), ls01 (candidate); a setting of one is rb spec set ls none. . ls declares it varies smoothing, but no variant has a value for it yet; give each variant its own: rb spec set ls .smoothing --source Next: rb spec set ls --from --keys "" · rb criterion add "" -e ls -q q-ls --metric --at-most ``` ```rb-session frame="terminal" $ rb spec set ls --from configs/train.yaml --keys "optim.*,data.*" ls: 3 settings from configs/train.yaml @d17af2687869 · 3 verified $ rb spec set ls none.smoothing 0.0 --source "configs/none.yaml#smoothing" ls/none.smoothing = 0.0 · verified in configs/none.yaml#smoothing $ rb spec set ls ls01.smoothing 0.1 --source "configs/ls01.yaml#smoothing" ls/ls01.smoothing = 0.1 · verified in configs/ls01.yaml#smoothing $ rb spec set ls seed --per-run ls/seed · per-run: each run gives its own, with --set seed= on rb run add ``` ```rb-session frame="terminal" $ rb metric add val_acc --unit % --maximize Metric val_acc (%): higher is better ``` 3. Record every row, one command per arm `--where smoothing=0.0` picks one arm's four rows, and `--every-row` records each as its own run. `--set seed=@seed` and `--produced-at @finished_at` read each run's seed and time from its row. A row already recorded is skipped. ```rb-session frame="terminal" $ rb run add ls --from results.tsv --where smoothing=0.0 --variant none --every-row --set seed=@seed --produced-at @finished_at runvnzbsp on ls: none.val_acc=71.42 · seed=1 runx7e5ci on ls: none.val_acc=71.18 · seed=2 rune332ux on ls: none.val_acc=71.55 · seed=3 runjh7ufm on ls: none.val_acc=71.3 · seed=4 4 of the 4 rows --where picks in results.tsv recorded as runs. rb put the code in git at refs/rb/snapshots/2f2ced2447df1446e5704536fc328b9bc50e2bb1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/2f2ced2447df1446e5704536fc328b9bc50e2bb1 removes it. $ rb run add ls --from results.tsv --where smoothing=0.1 --variant ls01 --every-row --set seed=@seed --produced-at @finished_at runm66q7u on ls: ls01.val_acc=72.06 · seed=1 runr3l6sg on ls: ls01.val_acc=71.87 · seed=2 runjrhrjq on ls: ls01.val_acc=72.21 · seed=3 runxoh3jt on ls: ls01.val_acc=71.95 · seed=4 4 of the 4 rows --where picks in results.tsv recorded as runs. ``` A `.jsonl` or `.ndjson` file works the same way, one JSON object a row. So does a SQLite table, with `--from results.db --table evals`. 4. Check the report's claim ```rb-session frame="terminal" $ rb compare ls ls Label smoothing 0.1 vs none · varies smoothing · 8 runs under the current spec · means over seeds variant role smoothing val_acc (%) none baseline 0.0 71.36 (n=4) ls01 candidate 0.1 72.02 +0.66 better (n=4) ``` ```rb-session frame="terminal" $ rb criterion add "Label smoothing 0.1 raises validation accuracy by about 0.66 points" -e ls -q q-ls --metric change.val_acc --equals 0.66 --tolerance 0.005 --source RESULTS.md --quote "Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds." --id ls-gain-claim ls-gain-claim: Label smoothing 0.1 raises validation accuracy by about 0.66 points change.val_acc = 0.66 ± 0.005 · primary · on ls cited: RESULTS.md "Label smoothing 0.1 raises validation accuracy by about 0..." · states 0.66 (read 'Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.') declared after the evidence existed: all 4 units were produced before it was declared it already reads change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb): reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005 Next: rb run add ls --from --set seed= ``` ```rb-session frame="terminal" $ rb show ls-gain-claim Criterion ls-gain-claim on ls (primary, cited): Label smoothing 0.1 raises validation accuracy by about 0.66 points cited: RESULTS.md "Label smoothing 0.1 raises validation accuracy by about 0..." · states 0.66, read 'Label smoothing 0.1 raises validation accuracy by about 0.66 points over four seeds.' ls-gain-claim Label smoothing 0.1 raises validation accuracy by about 0.66 points reproduced on the estimate · gain · change.val_acc = 0.66 ± 0.005 reads: change.val_acc: +0.66 [+0.6256, +0.6944] % · paired difference, t interval, 4 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (higher is better) rule: reproduced on the estimate: estimate +0.66 within 0.66 ± 0.005 timing: declared after the evidence existed: all 4 units were produced before it was declared units: 4 seeds from 8 runs integrity: no run config was compared, as said below · paired by seed: each seed's ls01 run against the same seed's none run; unpaired, the same runs read +0.66 [+0.3946, +0.9254] % · unpaired difference, Welch interval, 8 runs, 95% (rb); whether runs that share a seed are linked is not rb's to say. The repeat form is for a counter only (a seed, an epoch, a repeat's number), never a condition: where seed only numbers repeats, rb spec set ls population.repeat seed --no-verify reads each run as its own unit · no run config was compared with the spec (record runs with --config) · the interval [+0.6256, +0.6944] crosses 0.655 and 0.665: reproduced on the estimate, not across the whole interval runvnzbsp: none.val_acc 71.42 (row) · unit seed=1 · produced before the declaration · seed=1 runx7e5ci: none.val_acc 71.18 (row) · unit seed=2 · produced before the declaration · seed=2 rune332ux: none.val_acc 71.55 (row) · unit seed=3 · produced before the declaration · seed=3 runjh7ufm: none.val_acc 71.3 (row) · unit seed=4 · produced before the declaration · seed=4 runm66q7u: ls01.val_acc 72.06 (row) · unit seed=1 · produced before the declaration · seed=1 runr3l6sg: ls01.val_acc 71.87 (row) · unit seed=2 · produced before the declaration · seed=2 runjrhrjq: ls01.val_acc 72.21 (row) · unit seed=3 · produced before the declaration · seed=3 runxoh3jt: ls01.val_acc 71.95 (row) · unit seed=4 · produced before the declaration · seed=4 ``` 5. Read the repository again ```rb-session frame="terminal" $ rb reconcile The repository at commit d17af2687869: 2 results files in the top folder. Results files in no results folder (2): RESULTS.md: prose: rb reads no run from it; a number there is a cited rule a number it states is checked by a cited rule: rb criterion add "" -e --metric --equals --tolerance --source RESULTS.md --quote "" results.tsv: a TSV table of 8 rows, numbers in smoothing, seed, val_acc; 8 of 8 rows recorded as runs a row is a run: rb run add --from results.tsv --where =,... ``` --- # Record a folder per run https://rabbitbrain.ai/docs/cookbooks/run-folders/ rb records each run folder from what its config says, and points out a run whose config reports uncommitted code. ## Paste this into your agent **Record each run folder in results/ as a run, from what its own config says.** ```text Each folder in results/ is one run of eval.py. Record them with Rabbit Brain. - Run `rb context --model ` first, then `rb reconcile`. - Record the question and the experiment, with each setting cited to the config that states it, and the metric. - Run `rb reconcile` again, and record each folder with the `rb run add` line it prints, word for word. - Compute nothing yourself: rb computes the findings. Finish with `rb reconcile` and `rb status` exactly as printed. ``` The repository scores a small model on 500 grade-school maths problems with a plain prompt and with three worked examples, two seeds each. The fourth run's files: ```rb-session frame="terminal" $ cat results/r04/config.yaml command: python eval.py --config configs/examples.yaml --seed 2 --out results/r04 commit: 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc dirty: true started_at: 2026-10-28T11:30:00Z prompt: examples problems: 500 seed: 2 ``` ```rb-session frame="terminal" $ cat results/r04/metrics.json { "accuracy": 46.9, "finished_at": "2026-10-28T11:53:00Z" } ``` ## What you'll see `rb status` in your own terminal, once the agent is done: ```rb-session frame="terminal" $ rb status Worked examples in the prompt · worked-examples-in-the-prompt Kept with rb since 2026-11-10. All 4 runs were recorded afterwards, not as they happened. 13 record files not committed Do three worked examples in the prompt raise accuracy on grade-school maths? (open) No criterion is linked directly to this question. Worked examples vs a plain prompt · open Rules: none yet The design of 1 experiment is not confirmed: rb confirm lists it. Facts that ask for nothing (1) - examples: runs report examples.accuracy, plain.accuracy, which no recorded criterion reads rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 13 record files are new, changed or deleted since commit 855f30cb64ac, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/examples/experiment.yaml (new) and 10 more To check it yourself, in your terminal: git -C ~/prompt-eval status --short --untracked-files=all -- .rb Experiments examples Worked examples vs a plain prompt · open · plain (baseline) vs examples (candidate) · varies prompt · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 4 to run again: code partial · environment unknown · files ok Runs: 4 recorded on 1 experiment · every receipt is what rb wrote Next: rb confirm ``` - "environment unknown": rb was not watching when these runs were made, so it cannot say which machine made them. - A criterion is a rule a result must meet. None is recorded yet: no file here states what counts. ## What rb did, step by step 1. Read the repository A folder with a run's own config beside a results file is one run. `rb reconcile` prints an `rb run add` line for each, and writes nothing. Until an experiment is recorded, the line reads ``. ```rb-session frame="terminal" $ rb reconcile The repository at commit 855f30cb64ac: 4 results folders. 4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add --from records a run from one. Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw: results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4) rb run add '' --from results/r01/metrics.json --config results/r01/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1 results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4) rb run add '' --from results/r02/metrics.json --config results/r02/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1 results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4) rb run add '' --from results/r03/metrics.json --config results/r03/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1 results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4) results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran rb run add '' --from results/r04/metrics.json --config results/r04/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1 Settings that differ between them (2): prompt: plain (r01, r03), examples (r02, r04) seed: 1 (r01, r02), 2 (r03, r04) ``` 2. Record the question, experiment and settings ```rb-session frame="terminal" $ rb question add "Do three worked examples in the prompt raise accuracy on grade-school maths?" --id q-examples q-examples: Do three worked examples in the prompt raise accuracy on grade-school maths? Next: rb experiment add "" -q q-examples --hypothesis "" ``` ```rb-session frame="terminal" $ rb experiment add "Worked examples vs a plain prompt" --id examples -q q-examples --baseline plain --candidate examples --varies prompt Experiment examples: Worked examples vs a plain prompt · plain (baseline) vs examples (candidate) · varies prompt · under q-examples Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 2 variants: plain (baseline), examples (candidate); a setting of one is rb spec set examples plain. . examples declares it varies prompt, but no variant has a value for it yet; give each variant its own: rb spec set examples .prompt --source Next: rb spec set examples --from --keys "" · rb criterion add "" -e examples -q q-examples --metric --at-most ``` ```rb-session frame="terminal" $ rb spec set examples plain.prompt plain --source "configs/plain.yaml#prompt" examples/plain.prompt = plain · verified in configs/plain.yaml#prompt $ rb spec set examples examples.prompt examples --source "configs/examples.yaml#prompt" examples/examples.prompt = examples · verified in configs/examples.yaml#prompt $ rb spec set examples problems 500 --source "configs/plain.yaml#problems" examples/problems = 500 · verified in configs/plain.yaml#problems $ rb spec set examples seed --per-run examples/seed · per-run: each run gives its own, with --set seed= on rb run add ``` ```rb-session frame="terminal" $ rb metric add accuracy --unit % --maximize Metric accuracy (%): higher is better ``` 3. Read the folders again ```rb-session frame="terminal" $ rb reconcile The repository at commit 855f30cb64ac: 4 results folders. 4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add --from records a run from one. Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw: results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4) rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1 results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4) rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1 results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4) rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1 results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4) results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1 Settings that differ between them (2): prompt: plain (r01, r03), examples (r02, r04) seed: 1 (r01, r02), 2 (r03, r04) ``` 4. Record each folder ```rb-session frame="terminal" $ rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command "python eval.py --config configs/plain.yaml --seed 1 --out results/r01" --command-source results/r01/config.yaml:1 run3yui3v on examples: plain.accuracy=41.2 · seed=1 Ran with the spec: 3 of its settings found in results/r01/config.yaml, all matching. Produced: 2026-10-28 10:20:00 (reported) rb put the code in git at refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1 removes it. Since this session last looked: - 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/r02/, results/r03/, results/r04/ (rb reconcile) Next: rb criterion add "" -e examples -q q-examples --metric plain.accuracy --at-most (write the criterion before the next run) $ rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command "python eval.py --config configs/examples.yaml --seed 1 --out results/r02" --command-source results/r02/config.yaml:1 runqtkwtb on examples: examples.accuracy=47.8 · seed=1 Ran with the spec: 3 of its settings found in results/r02/config.yaml, all matching. Produced: 2026-10-28 10:52:00 (reported) Since this session last looked: - 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/r03/, results/r04/ (rb reconcile) Next: rb criterion add "" -e examples -q q-examples --metric examples.accuracy --at-most (write the criterion before the next run) $ rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command "python eval.py --config configs/plain.yaml --seed 2 --out results/r03" --command-source results/r03/config.yaml:1 runsoyo2w on examples: plain.accuracy=40.6 · seed=2 Ran with the spec: 3 of its settings found in results/r03/config.yaml, all matching. Produced: 2026-10-28 11:21:00 (reported) Since this session last looked: - results/r04/ holds results written before the record began, which no recorded experiment reads yet (rb reconcile) Next: rb criterion add "" -e examples -q q-examples --metric plain.accuracy --at-most (write the criterion before the next run) $ rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command "python eval.py --config configs/examples.yaml --seed 2 --out results/r04" --command-source results/r04/config.yaml:1 runfvajdy on examples: examples.accuracy=46.9 · seed=2 Ran with the spec: 3 of its settings found in results/r04/config.yaml, all matching. results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran. Produced: 2026-10-28 11:53:00 (reported) Next: rb criterion add "" -e examples -q q-examples --metric examples.accuracy --at-most (write the criterion before the next run) ``` 5. Read the repository again ```rb-session frame="terminal" $ rb reconcile The repository at commit 855f30cb64ac: 4 results folders. Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw: results/r01/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4) recorded as run3yui3v: rb show run3yui3v results/r02/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4) recorded as runqtkwtb: rb show runqtkwtb results/r03/ config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4) recorded as runsoyo2w: rb show runsoyo2w results/r04/ config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4) results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran recorded as runfvajdy: rb show runfvajdy Settings that differ between them (2): prompt: plain (r01, r03), examples (r02, r04) seed: 1 (r01, r02), 2 (r03, r04) results/r01/ read by the runs of examples files: 2 results files git: both report and results files are tracked · folder unchanged since the last commit results/r02/ read by the runs of examples files: 2 results files git: both report and results files are tracked · folder unchanged since the last commit results/r03/ read by the runs of examples files: 2 results files git: both report and results files are tracked · folder unchanged since the last commit results/r04/ read by the runs of examples files: 2 results files git: both report and results files are tracked · folder unchanged since the last commit ``` --- # Check a summary's claims against the runs https://rabbitbrain.ai/docs/cookbooks/check-a-summary/ Check an agent's summary before you repeat it to your team. Each claim becomes a criterion, a rule rb checks against the runs. ## Paste this into your agent **Check every claim in SUMMARY.md against the runs, with rb doing the reading.** ```text My last session wrote SUMMARY.md. Check each claim in it against the runs with Rabbit Brain, and take nothing on the summary's word. - Run `rb context --model ` first. - Record the experiment, and every run from its own file in `results/`. - Turn each claim into the criterion it states, cited to its line: `--source SUMMARY.md --quote ""`. Use the claim's own number, with no threshold of yours. - A claim no criterion can read goes in a note that names the files you read. - Compute nothing yourself: rb computes the findings. Finish with `rb status` exactly as printed, and for each claim, what rb says of it. ``` We ran it on an INT8 ResNet-20 repository with three seeds and this summary: ```markdown title="SUMMARY.md" # Session summary - INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed. - INT8 averages 91.62 top-1 over the three seeds. - 512 calibration images are enough. ``` ## What you'll see ```rb-session frame="terminal" $ rb status INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10 Kept with rb since 2026-11-10. All 3 runs were recorded afterwards, not as they happened. 15 record files not committed Does INT8 match FP32 top-1 on CIFAR-10? (open) On every int8-ptq criterion below: question link declared after the evidence existed: all 3 units were produced before it was declared; units: 3 seeds. The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here. top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2); rule and question link declared after the evidence existed: all 3 units were produced before it was declared. The summary's claim that INT8 averages 91.62 top-1 is checked here. int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; reproduced on the estimate: estimate 91.6233 within 91.62 ± 0.005; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [91.48, 91.77] crosses 91.615 and 91.625: reproduced on the estimate, not across the whole interval. experiment context: int8-ptq · INT8 post-training quantization note nsctsjr: No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/. · not confirmed Rules: 1 reproduced, on the estimate alone · 1 not reproduced Needs judgment (1) - every-seed-claim is not reproduced and no call on it is recorded: The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here rb decide every-seed-claim accept|reject|investigate --why "..." The design of 1 experiment is not confirmed: rb confirm lists it. Facts that ask for nothing (1) - int8-ptq: 2 criteria were declared after their evidence existed rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 15 record files are new, changed or deleted since commit 6eed520f0d01, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/int8-ptq/criteria/every-seed-claim.yaml (new) and 12 more To check it yourself, in your terminal: git -C ~/int8-summary status --short --untracked-files=all -- .rb Rules every-seed-claim The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here not reproduced · gain · each unit's top1: fp32 vs int8 ≤ 0.05 reads: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (higher is better) rule: not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2) timing: declared after the evidence existed: all 3 units were produced before it was declared units: 3 seeds int8-mean-claim The summary's claim that INT8 averages 91.62 top-1 is checked here reproduced on the estimate · measured · int8.top1 = 91.62 ± 0.005 reads: int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb) result: measured: int8 on its own, with its interval, so neither better nor worse rule: reproduced on the estimate: estimate 91.6233 within 91.62 ± 0.005 timing: declared after the evidence existed: all 3 units were produced before it was declared units: 3 seeds · the interval [91.48, 91.77] crosses 91.615 and 91.625: reproduced on the estimate, not across the whole interval Experiments int8-ptq INT8 post-training quantization · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 3 to run again: code unknown · environment unknown · files ok Runs: 3 recorded on 1 experiment · every receipt is what rb wrote Next: rb confirm · rb decide every-seed-claim accept|reject|investigate --why "..." ``` - Seeds 1 and 2 break the "never loses more than 0.05" claim. The criterion compares FP32 against INT8, so "gain" means FP32 is ahead. - The 91.62 average is reproduced on the estimate, not across the whole interval. - No run compares calibration sizes, so the 512-images claim gets a note, not a criterion. The 0.05 claim waits under **Needs judgment**. The call on it, a decision you record with `rb decide`, is yours, not the agent's. ## What rb did, step by step 1. Record the experiment and its settings ```rb-session frame="terminal" $ rb experiment add "INT8 post-training quantization" --id int8-ptq -q q-int8 --baseline fp32 --candidate int8 --varies precision,calib_images Experiment int8-ptq: INT8 post-training quantization · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · under q-int8 Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 2 variants: fp32 (baseline), int8 (candidate); a setting of one is rb spec set int8-ptq fp32. . int8-ptq declares it varies precision, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq .precision --source int8-ptq declares it varies calib_images, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq .calib_images --source Next: rb spec set int8-ptq --from --keys "" · rb criterion add "" -e int8-ptq -q q-int8 --metric --at-most ``` ```rb-session frame="terminal" $ rb spec set int8-ptq fp32.precision fp32 --source "configs/fp32.yaml#precision" int8-ptq/fp32.precision = fp32 · verified in configs/fp32.yaml#precision $ rb spec set int8-ptq int8.precision int8 --source "configs/int8.yaml#precision" int8-ptq/int8.precision = int8 · verified in configs/int8.yaml#precision $ rb spec set int8-ptq int8.calib_images 512 --source "configs/int8.yaml#calib_images" int8-ptq/int8.calib_images = 512 · verified in configs/int8.yaml#calib_images $ rb spec set int8-ptq seed --per-run int8-ptq/seed · per-run: each run gives its own, with --set seed= on rb run add ``` 2. Turn each claim into the criterion it states With `--source`, rb checks that the quoted line is there and holds the criterion's number. That is why the 0.05 claim is written as FP32 against INT8. ```rb-session frame="terminal" $ rb criterion add "The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here" -e int8-ptq -q q-int8 --metric top1 --subject fp32 --reference int8 --at-most 0.05 --each --source SUMMARY.md --quote "it never loses more than 0.05 points of top-1 on any seed" --id every-seed-claim every-seed-claim: The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here each unit's top1: fp32 vs int8 ≤ 0.05 · primary · on int8-ptq cited: SUMMARY.md "it never loses more than 0.05 points of top-1 on any seed" · states 0.05 (read '- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.') declared; no evidence recorded yet Next: rb run add int8-ptq --from --set seed= $ rb criterion add "The summary's claim that INT8 averages 91.62 top-1 is checked here" -e int8-ptq -q q-int8 --metric int8.top1 --equals 91.62 --tolerance 0.005 --source SUMMARY.md --quote "INT8 averages 91.62 top-1 over the three seeds" --id int8-mean-claim int8-mean-claim: The summary's claim that INT8 averages 91.62 top-1 is checked here int8.top1 = 91.62 ± 0.005 · primary · on int8-ptq cited: SUMMARY.md "INT8 averages 91.62 top-1 over the three seeds" · states 91.62 (read '- INT8 averages 91.62 top-1 over the three seeds.') declared; no evidence recorded yet Next: rb run add int8-ptq --from --set seed= ``` 3. Record each run from its own file After the third seed: ```rb-session frame="terminal" $ rb run add int8-ptq --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml runpcz6en on int8-ptq: fp32.top1=91.74, int8.top1=91.69 · seed=3 Ran with the spec: 1 of its settings found in results/int8-ptq/seed3/config.yaml, matching. Produced: 2026-10-20 10:00:00 (file) every-seed-claim not reproduced · gain: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb) int8-mean-claim reproduced on the estimate · measured: int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb) Next: rb show every-seed-claim · rb show int8-mean-claim ``` 4. Note the claim no criterion can read ```rb-session frame="terminal" $ rb note add "No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/." -e int8-ptq Note nsctsjr on int8-ptq: No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/. Kept verbatim with who wrote it; a note changes no finding. ``` 5. Read each claim's criterion ```rb-session frame="terminal" $ rb show every-seed-claim Criterion every-seed-claim on int8-ptq (primary, cited): The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here cited: SUMMARY.md "it never loses more than 0.05 points of top-1 on any seed" · states 0.05, read '- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.' every-seed-claim The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here not reproduced · gain · each unit's top1: fp32 vs int8 ≤ 0.05 reads: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (higher is better) rule: not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2) timing: declared after the evidence existed: all 3 units were produced before it was declared units: 3 seeds integrity: ok runikex4f: difference +0.11 (file) · unit seed=1 · produced before the declaration · seed=1 rungwaqdb: difference +0.08 (file) · unit seed=2 · produced before the declaration · seed=2 runpcz6en: difference +0.05 (file) · unit seed=3 · produced before the declaration · seed=3 ``` --- # Declare the criterion before the runs https://rabbitbrain.ai/docs/cookbooks/criterion-before-runs/ A threshold picked after the numbers came in was fitted to them. Declare the criterion, the rule a result must meet, and freeze it before the first run. ## Paste this into your agent **Declare the criterion and freeze it before any run, then run the seeds.** ```text Use Rabbit Brain to find out if warmup is worth it for the 6-layer GPT. docs/plan.md says what counts. - Run `rb context --model ` first. - Before any run: record the question, the experiment with both arms and every setting cited to its config, and the criterion docs/plan.md states, cited to its line. Then freeze the experiment. - Then run seeds 1 to 3 through `rb journal run -- python train.py --seed `, and record each run from its output file, with its config. - Compute nothing yourself. Finish with `rb show` of the criterion exactly as printed. ``` ## What you'll see After three seeds: ```rb-session frame="terminal" $ rb show warmup-helps Criterion warmup-helps on warmup (primary, own): Warmup lowers final validation loss by at least 0.02 nats warmup-helps Warmup lowers final validation loss by at least 0.02 nats criterion met · gain · change.val_loss ≤ -0.02 (n ≥ 3) reads: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb) result: gain: the interval clear of 0 on the better side (lower is better) rule: criterion met: estimate -0.02767 ≤ -0.02 timing: declared before the evidence (by production times) units: 3 seeds integrity: ok runttgab7: difference -0.031 (file) · unit seed=1 · seed=1 runvw3726: difference -0.026 (file) · unit seed=2 · seed=2 run2ugudg: difference -0.026 (file) · unit seed=3 · seed=3 ``` The timing line reads **declared before the evidence**. Raising the bar to 0.03 nats now needs a reason, because the experiment is frozen: ```rb-session frame="terminal" $ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar rb criterion add: Experiment warmup was frozen at 2026-11-10T15:32:13.795+00:00 (e84183e9a687); this changes it (new criterion stricter-warmup-bar: change.val_loss ≤ -0.03 (n ≥ 3)). → Give the change its reason: the same command with --amend --why "". The amendment is recorded, by whoever makes it. $ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar --amend --why "a stricter bar, set after seeing the runs" stricter-warmup-bar: Warmup lowers final validation loss by at least 0.03 nats change.val_loss ≤ -0.03 (n ≥ 3) · primary · on warmup declared after the evidence existed: all 3 units were produced before it was declared it already reads change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb): criterion not met: estimate -0.02767 > -0.03 Next: rb run add warmup --from --set seed= ``` `rb status` keeps the two criteria apart by when each was declared: ```rb-session frame="terminal" $ rb status Warmup on a small GPT · warmup-on-a-small-gpt 14 record files not committed Is warmup worth it for the 6-layer GPT? (open) On every warmup criterion below: units: 3 seeds. Warmup lowers final validation loss by at least 0.03 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion not met: estimate -0.02767 > -0.03; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [-0.03484, -0.0205] crosses -0.03: missed on the estimate, not across the whole interval. Warmup lowers final validation loss by at least 0.02 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion met: estimate -0.02767 ≤ -0.02; rule and question link declared before the evidence (by production times). Rules: 1 met · 1 not met Needs judgment (1) - stricter-warmup-bar misses its rule and no call on it is recorded: Warmup lowers final validation loss by at least 0.03 nats rb decide stricter-warmup-bar accept|reject|investigate --why "..." The design of 1 experiment is not confirmed: rb confirm lists it. Can be done now (1) - warmup: 1 criterion was declared after its evidence existed: a reading of its own needs its subject's runs made after the declaration rb run add warmup --from --set seed= (run after the declaration) rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 14 record files are new, changed or deleted since commit 90e2f71dfd90, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/warmup/criteria/stricter-warmup-bar.yaml (new) and 11 more To check it yourself, in your terminal: git -C ~/warmup-gpt status --short --untracked-files=all -- .rb … (25 more lines) ``` The late criterion reads **declared after the evidence existed**. Its miss waits for your call, a decision you record with `rb decide`. See [Calls](https://rabbitbrain.ai/docs/concepts/calls/). ## What rb did, step by step 1. Record the experiment and cite its settings ```rb-session frame="terminal" $ rb experiment add "Warmup vs no warmup" --id warmup -q q-warmup --baseline none --candidate warmup --varies warmup_steps Experiment warmup: Warmup vs no warmup · none (baseline) vs warmup (candidate) · varies warmup_steps · under q-warmup Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 2 variants: none (baseline), warmup (candidate); a setting of one is rb spec set warmup none. . warmup declares it varies warmup_steps, but no variant has a value for it yet; give each variant its own: rb spec set warmup .warmup_steps --source Next: rb spec set warmup --from --keys "" · rb criterion add "" -e warmup -q q-warmup --metric --at-most ``` ```rb-session frame="terminal" $ rb spec set warmup --from configs/base.yaml --keys "optim.*,data.*" warmup: 4 settings from configs/base.yaml @90e2f71dfd90 · 4 verified $ rb spec set warmup none.warmup_steps 0 --source "configs/no-warmup.yaml#warmup_steps" warmup/none.warmup_steps = 0 · verified in configs/no-warmup.yaml#warmup_steps $ rb spec set warmup warmup.warmup_steps 500 --source "configs/warmup.yaml#warmup_steps" warmup/warmup.warmup_steps = 500 · verified in configs/warmup.yaml#warmup_steps $ rb spec set warmup seed --per-run warmup/seed · per-run: each run gives its own, with --set seed= on rb run add ``` 2. Declare the plan's criterion With `--why "docs/plan.md:3"`, rb checks that the line is there and states the threshold. ```rb-session frame="terminal" $ rb criterion add "Warmup lowers final validation loss by at least 0.02 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.02 --min-n 3 --why docs/plan.md:3 --id warmup-helps warmup-helps: Warmup lowers final validation loss by at least 0.02 nats change.val_loss ≤ -0.02 (n ≥ 3) · primary · on warmup declared; no evidence recorded yet Next: rb run add warmup --from --set seed= ``` 3. Freeze it ```rb-session frame="terminal" $ rb freeze warmup -m "before the runs" warmup frozen: spec e84183e9a687 · before the runs warmup: Warmup vs no warmup variants: none (baseline), warmup (candidate) · varies warmup_steps criterion warmup-helps (primary): change.val_loss ≤ -0.02 (n ≥ 3) optim.lr = 0.0006 · verified · configs/base.yaml#optim.lr optim.weight_decay = 0.1 · verified · configs/base.yaml#optim.weight_decay data.batch_size = 64 · verified · configs/base.yaml#data.batch_size data.seq_len = 256 · verified · configs/base.yaml#data.seq_len seed = (each run) · per-run · no source none.warmup_steps = 0 · verified · configs/no-warmup.yaml#warmup_steps warmup.warmup_steps = 500 · verified · configs/warmup.yaml#warmup_steps look: warmup-helps is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "". Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why ""; the amendment is kept with who made it. Next: rb run add warmup --from --command "..." ``` 4. Run each seed ```rb-session frame="terminal" $ rb journal run -- python train.py --seed 1 seed 1: val_loss none 2.412 warmup 2.381 {"metrics": {"none.val_loss": 2.412, "warmup.val_loss": 2.381}} exit 0. The command, its exit code and what it wrote are in the journal. It printed its metrics: none.val_loss=2.412, warmup.val_loss=2.381. rb reconcile proposes the run of warmup that records them. Since this session last looked: - out/seed1/ holds results no recorded experiment reads yet (rb reconcile) - out/seed1/metrics.json: results landed and are not in the record (rb reconcile) Next: rb reconcile ``` 5. Record each run ```rb-session frame="terminal" $ rb run add warmup --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1" runttgab7 on warmup: none.val_loss=2.412, warmup.val_loss=2.381 · seed=1 Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching. Produced: 2026-11-10 15:32:13 (file) warmup-helps criterion undetermined · point only: change.val_loss: -0.031 nats · paired difference, 1 seed (rb) rb put the code in git at refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54 removes it. Next: rb show warmup-helps $ rb run add warmup --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2" runvw3726 on warmup: none.val_loss=2.405, warmup.val_loss=2.379 · seed=2 Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching. Produced: 2026-11-10 15:32:14 (file) warmup-helps criterion undetermined · point only: change.val_loss: -0.0285 nats · paired difference, 2 seeds (rb) Next: rb show warmup-helps $ rb run add warmup --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3" run2ugudg on warmup: none.val_loss=2.418, warmup.val_loss=2.392 · seed=3 Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching. Produced: 2026-11-10 15:32:14 (file) warmup-helps criterion met · gain: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show warmup-helps ``` --- # Compare against a published baseline https://rabbitbrain.ai/docs/cookbooks/published-baseline/ A match to a paper counts only on the paper's settings. Cite its number as a criterion, a rule a result must meet, and cite its settings. ## Paste this into your agent **Check our run against the paper's number, with the paper's settings cited.** ```text Check our ResNet-20 against the paper's 8.75% error with Rabbit Brain. The paper's numbers are saved in docs/baselines/resnet-paper.md. Count it as reproduced if we land within 0.3 points of the paper's 8.75% error. - Run `rb context --model ` first. - Record the experiment, and each training setting from configs/train.yaml with the value the paper used, cited to its line in docs/baselines/resnet-paper.md (`--cited`, `--cited-source`). - Record the paper's number as a criterion cited to its table row (`--source`, `--quote`, `--locator "Table 6"`), with my tolerance and my words. - Run seeds 1 to 3 through `rb journal run`, and record each run. Finish with `rb show` of the criterion exactly as printed, and every setting that differs from the paper's. ``` The paper's text is saved in `docs/baselines/resnet-paper.md`: ```markdown title="docs/baselines/resnet-paper.md" # Deep Residual Learning for Image Recognition (2016): excerpts Section 4.2, CIFAR-10: We use a weight decay of 0.0001 and momentum of 0.9. These models are trained with a mini-batch size of 128. We start with a learning rate of 0.1. Table 6, classification error on the CIFAR-10 test set: | method | layers | params | error (%) | |---|---|---|---| | ResNet | 20 | 0.27M | 8.75 | | ResNet | 32 | 0.46M | 7.51 | ``` ## What you'll see ```rb-session frame="terminal" $ rb show matches-paper-error Criterion matches-paper-error on repro (primary, cited): Our ResNet-20 lands within 0.3 points of the paper's 8.75% error · words (as given): "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error." cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75, read '| ResNet | 20 | 0.27M | 8.75 |' matches-paper-error Our ResNet-20 lands within 0.3 points of the paper's 8.75% error reproduced · measured · resnet20.error = 8.75 ± 0.3 (n ≥ 3) reads: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb) result: measured: resnet20 on its own, with its interval, so neither better nor worse rule: reproduced: estimate 8.747 within 8.75 ± 0.3 timing: declared before the evidence (by production times) units: 3 seeds integrity: ok rune5stgq: resnet20.error 8.81 (file) · unit seed=1 · seed=1 runx5twzo: resnet20.error 8.69 (file) · unit seed=2 · seed=2 runrvxoet: resnet20.error 8.74 (file) · unit seed=3 · seed=3 ``` ## What rb did, step by step 1. Record the experiment's purpose ```rb-session frame="terminal" $ rb experiment add "ResNet-20 against the ResNet paper (2016)" --id repro -q q-repro --purpose "Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6" --baseline resnet20 Experiment repro: ResNet-20 against the ResNet paper (2016) · resnet20 (baseline) · under q-repro purpose: Reproduce the ResNet-20 CIFAR-10 test error of the ResNet paper (2016), Table 6 Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis. Made 1 variant: resnet20 (baseline); a setting of one is rb spec set repro resnet20. . Next: rb spec set repro --from --keys "" · rb criterion add "" -e repro -q q-repro --metric --at-most ``` 2. Cite each setting and the paper's value `--cited` keeps the paper's value beside yours. A difference makes every criterion that cites the paper not comparable. ```rb-session frame="terminal" $ rb spec set repro optim.lr --source "configs/train.yaml#optim.lr" --cited 0.1 --cited-source docs/baselines/resnet-paper.md:7 repro/optim.lr = 0.1 · verified in configs/train.yaml#optim.lr cited: 0.1 (docs/baselines/resnet-paper.md:7) $ rb spec set repro optim.momentum --source "configs/train.yaml#optim.momentum" --cited 0.9 --cited-source docs/baselines/resnet-paper.md:5 repro/optim.momentum = 0.9 · verified in configs/train.yaml#optim.momentum cited: 0.9 (docs/baselines/resnet-paper.md:5) $ rb spec set repro optim.weight_decay --source "configs/train.yaml#optim.weight_decay" --cited 0.0001 --cited-source docs/baselines/resnet-paper.md:5 repro/optim.weight_decay = 0.0001 · verified in configs/train.yaml#optim.weight_decay cited: 0.0001 (docs/baselines/resnet-paper.md:5) $ rb spec set repro data.batch_size --source "configs/train.yaml#data.batch_size" --cited 128 --cited-source docs/baselines/resnet-paper.md:6 repro/data.batch_size = 128 · verified in configs/train.yaml#data.batch_size cited: 128 (docs/baselines/resnet-paper.md:6) $ rb spec set repro seed --per-run repro/seed · per-run: each run gives its own, with --set seed= on rb run add ``` ```rb-session frame="terminal" $ rb show repro/data.batch_size repro/data.batch_size = 128 · verified source: configs/train.yaml#data.batch_size · read 'data.batch_size: 128' @9d87bd151701 cited source used 128 (docs/baselines/resnet-paper.md:6) history: 2026-11-10T15:32:17 spec set: setting data.batch_size: new, 128 2026-11-10T15:32:17 spec check: verified ``` 3. Cite the paper's number as a criterion rb checks that the quoted row is in the file and states 8.75. ```rb-session frame="terminal" $ rb criterion add "Our ResNet-20 lands within 0.3 points of the paper's 8.75% error" -e repro -q q-repro --metric resnet20.error --equals 8.75 --tolerance 0.3 --source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6" --min-n 3 --words "Count it as reproduced if we land within 0.3 points of the paper's 8.75% error." --id matches-paper-error matches-paper-error: Our ResNet-20 lands within 0.3 points of the paper's 8.75% error resnet20.error = 8.75 ± 0.3 (n ≥ 3) · primary · on repro cited: docs/baselines/resnet-paper.md "| ResNet | 20 | 0.27M | 8.75 |" (Table 6) · states 8.75 (read '| ResNet | 20 | 0.27M | 8.75 |') declared; no evidence recorded yet Next: rb run add repro --from --set seed= ``` 4. Run and record each seed ```rb-session frame="terminal" $ rb journal run -- python train.py --seed 1 {"metrics": {"resnet20.error": 8.81}} exit 0. The command, its exit code and what it wrote are in the journal. It printed its metrics: resnet20.error=8.81. rb reconcile proposes the run of repro that records them. Since this session last looked: - out/seed1/ holds results no recorded experiment reads yet (rb reconcile) - out/seed1/metrics.json: results landed and are not in the record (rb reconcile) Next: rb reconcile ``` ```rb-session frame="terminal" $ rb run add repro --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1" rune5stgq on repro: resnet20.error=8.81 · seed=1 Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching. Produced: 2026-11-10 15:32:17 (file) matches-paper-error criterion undetermined · point only: resnet20.error: 8.81 % · mean, 1 seed (rb) rb put the code in git at refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/e8626933b5fe8d9b4040088d8db7c87428d15d30 removes it. Next: rb show matches-paper-error $ rb run add repro --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2" runx5twzo on repro: resnet20.error=8.69 · seed=2 Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching. Produced: 2026-11-10 15:32:17 (file) matches-paper-error criterion undetermined · point only: resnet20.error: 8.75 % · mean, 2 seeds (rb) Next: rb show matches-paper-error $ rb run add repro --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3" runrvxoet on repro: resnet20.error=8.74 · seed=3 Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching. Produced: 2026-11-10 15:32:18 (file) matches-paper-error reproduced · measured: resnet20.error: 8.747 [8.597, 8.896] % · mean, t interval, 3 seeds, 95% (rb) Next: rb show matches-paper-error ``` --- # Get an interval, not a single number https://rabbitbrain.ai/docs/cookbooks/interval-from-seeds/ With one seed you cannot tell a real cost from run-to-run noise. Ask for an interval in the criterion, the rule a result must meet, and set how many seeds it needs. ## Paste this into your agent **Read the result on its 95% interval, over 5 seeds declared up front.** ```text Use Rabbit Brain to find out whether dropout 0.1 costs us validation loss on the MLP. docs/plan.md says what counts. I want an interval, not one number: dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout. - Run `rb context --model ` first. - Before any run: record the experiment, its settings from the configs, the plan's criterion cited to its line, and my criterion on the interval, with my words, needing 5 seeds. Freeze it. - Run seeds 1 to 5 through `rb journal run`, and record each run. - After each seed, tell me rb's finding as printed. Never round "point only" up to a result. Finish with `rb show` of both criteria exactly as printed. ``` Two criteria read dropout's validation loss minus no dropout's: - `mean-within-budget` is the plan's: the mean cost is at most 0.02 nats. - `interval-within-budget` is yours: the whole 95% interval stays within 0.02 nats, on at least 5 seeds. ## What you'll see After each seed, `rb run add` prints both findings, what rb reads for each criterion from the runs: | Seeds | Reading | `mean-within-budget` | `interval-within-budget` | | --- | --- | --- | --- | | 1 | +0.008 nats | estimate meets criterion · point only | criterion undetermined · point only | | 2 | +0.0025 nats | estimate meets criterion · point only | criterion undetermined · point only | | 3 | +0.006333 [-0.01508, +0.02775] nats | estimate meets criterion · unresolved | criterion undetermined · unresolved | | 4 | +0.0055 [-0.006011, +0.01701] nats | criterion met · unresolved | criterion undetermined · unresolved | | 5 | +0.0056 [-0.002184, +0.01338] nats | criterion met · unresolved | criterion met · unresolved | At three seeds the interval reaches past 0.02. Your criterion stays undetermined until its fifth seed. ```rb-session frame="terminal" $ rb status Dropout on a small MLP · dropout-on-a-small-mlp 16 record files not committed Does dropout 0.1 cost us validation loss on the MLP? (open) On every drop criterion below: question link declared before the evidence (by production times); units: 5 seeds. The whole 95% interval of dropout's cost stays within 0.02 nats. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: the interval's worse bound +0.01338 inside 0.02; rule and question link declared before the evidence (by production times) · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout". Dropout 0.1 costs at most 0.02 nats of validation loss. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: estimate +0.0056 ≤ 0.02; rule and question link declared before the evidence (by production times). … (34 more lines) ``` **Unresolved** means the interval includes 0. It is not a failure: both criteria ask only for a cost within 0.02 nats. ## What rb did, step by step 1. Declare both criteria ```rb-session frame="terminal" $ rb criterion add "Dropout 0.1 costs at most 0.02 nats of validation loss" -e drop -q q-drop --metric change.val_loss --at-most 0.02 --why docs/plan.md:3 --id mean-within-budget mean-within-budget: Dropout 0.1 costs at most 0.02 nats of validation loss change.val_loss ≤ 0.02 · primary · on drop declared; no evidence recorded yet Next: rb run add drop --from --set seed= $ rb criterion add "The whole 95% interval of dropout's cost stays within 0.02 nats" -e drop -q q-drop --metric change.val_loss --noninferior 0.02 --min-n 5 --words "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" --id interval-within-budget interval-within-budget: The whole 95% interval of dropout's cost stays within 0.02 nats change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) · primary · on drop declared; no evidence recorded yet Next: rb run add drop --from --set seed= ``` 2. Freeze the experiment ```rb-session frame="terminal" $ rb freeze drop -m "before the first seed" drop frozen: spec 12f84c212b8c · before the first seed drop: Dropout 0.1 vs none variants: none (baseline), p01 (candidate) · varies dropout criterion mean-within-budget (primary): change.val_loss ≤ 0.02 criterion interval-within-budget (primary): change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) (words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout") optim.lr = 0.001 · verified · configs/train.yaml#optim.lr data.batch_size = 256 · verified · configs/train.yaml#data.batch_size seed = (each run) · per-run · no source none.dropout = 0.0 · verified · configs/none.yaml#dropout p01.dropout = 0.1 · verified · configs/p01.yaml#dropout look: mean-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "". look: interval-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "". Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why ""; the amendment is kept with who made it. Next: rb run add drop --from --command "..." ``` 3. Run and record each seed ```rb-session frame="terminal" $ rb journal run -- python train.py --seed 1 {"metrics": {"none.val_loss": 1.31, "p01.val_loss": 1.318}} exit 0. The command, its exit code and what it wrote are in the journal. It printed its metrics: none.val_loss=1.31, p01.val_loss=1.318. rb reconcile proposes the run of drop that records them. Since this session last looked: - out/seed1/ holds results no recorded experiment reads yet (rb reconcile) - out/seed1/metrics.json: results landed and are not in the record (rb reconcile) Next: rb reconcile ``` ```rb-session frame="terminal" $ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1" rungxyf6p on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1 Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching. Produced: 2026-11-10 15:32:22 (file) mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb) interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb) rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it. Next: rb show mean-within-budget · rb show interval-within-budget $ rb run add drop --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2" runbklxmd on drop: none.val_loss=1.302, p01.val_loss=1.299 · seed=2 Ran with the spec: 3 of its settings found in out/seed2/config.yaml, all matching. Produced: 2026-11-10 15:32:23 (file) mean-within-budget estimate meets criterion · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb) interval-within-budget criterion undetermined · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb) Next: rb show mean-within-budget · rb show interval-within-budget $ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3" runj5w6lb on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3 Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching. Produced: 2026-11-10 15:32:23 (file) mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb) interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show mean-within-budget · rb show interval-within-budget $ rb run add drop --from out/seed4/metrics.json --set seed=4 --config out/seed4/config.yaml --command "python train.py --seed 4" runrevodj on drop: none.val_loss=1.308, p01.val_loss=1.311 · seed=4 Ran with the spec: 3 of its settings found in out/seed4/config.yaml, all matching. Produced: 2026-11-10 15:32:24 (file) mean-within-budget criterion met · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb) interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb) Next: rb show mean-within-budget · rb show interval-within-budget $ rb run add drop --from out/seed5/metrics.json --set seed=5 --config out/seed5/config.yaml --command "python train.py --seed 5" rungtlduu on drop: none.val_loss=1.312, p01.val_loss=1.318 · seed=5 Ran with the spec: 3 of its settings found in out/seed5/config.yaml, all matching. Produced: 2026-11-10 15:32:24 (file) mean-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb) interval-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb) Next: rb show mean-within-budget · rb show interval-within-budget ``` 4. Read your criterion ```rb-session frame="terminal" $ rb show interval-within-budget Criterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) reads: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb) result: unresolved: the interval includes 0 rule: criterion met: the interval's worse bound +0.01338 inside 0.02 timing: declared before the evidence (by production times) units: 5 seeds integrity: ok rungxyf6p: difference +0.008 (file) · unit seed=1 · seed=1 runbklxmd: difference -0.003 (file) · unit seed=2 · seed=2 runj5w6lb: difference +0.014 (file) · unit seed=3 · seed=3 runrevodj: difference +0.003 (file) · unit seed=4 · seed=4 rungtlduu: difference +0.006 (file) · unit seed=5 · seed=5 ``` 5. Compare the arms ```rb-session frame="terminal" $ rb compare drop drop Dropout 0.1 vs none · varies dropout · 5 runs under the current spec · means over seeds variant role dropout val_loss (nats) none baseline 0.0 1.309 (n=5) p01 candidate 0.1 1.315 +0.0056 worse [primary] (n=5) the primary criteria read: val_loss p01 vs none (mean-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)), val_loss p01 vs none (interval-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)); deltas are means, the findings carry the intervals (rb show ) ``` --- # Read a rule over seeds https://rabbitbrain.ai/docs/cookbooks/rules-over-seeds/ "The mean meets 91.58" and "4 of 5 seeds meet 91.58" are different claims. rb checks the one your plan makes. ## Paste this into your agent **Record the five seeds, and the plan's three rules over them.** ```text Use Rabbit Brain to check INT8 against FP32 over the five seeds in results/int8-ptq/. docs/plan.md says what counts, one rule a line. - Run `rb context --model ` first. - Record the question and the experiment, with each setting cited to the config that states it. - Record each of the plan's three rules as rb reads it, cited to its line: a count (`--count 4 --of 5`), a best of N (`--best --of 5`) and a relative change (`--relative`). - Record each seed as a run. - Compute nothing yourself: rb computes the findings. Finish with `rb status` and `rb show` of each criterion exactly as printed. ``` The repository evaluates an FP32 ResNet-20 and its INT8 copy on CIFAR-10 for five seeds. Lines 4 to 6 of `docs/plan.md` hold the three rules. ## What you'll see `rb status` in your own terminal, once the agent is done: ```rb-session frame="terminal" $ rb status INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10 Kept with rb since 2026-11-10. All 5 runs were recorded afterwards, not as they happened. 17 record files not committed Does INT8 hold up against FP32 across the five seeds? (open) On every int8-seeds criterion below: question link declared after the evidence existed: all 5 units were produced before it was declared; units: 5 seeds. INT8's best of 5 seeds reaches 91.65% top-1. int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb); point only: a best of N is one unit's own value, with no interval; recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. INT8 reaches 91.58% top-1 on at least 4 of 5 seeds. int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. INT8 loses at most 0.1% of FP32's top-1. change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb); loss: the interval clear of 0 on the worse side (higher is better); estimate meets criterion: estimate -0.0916% ≥ -0.1%; rule and question link declared after the evidence existed: all 5 units were produced before it was declared. the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval. Rules: 3 met, all 3 on the estimate alone The design of 1 experiment is not confirmed: rb confirm lists it. Facts that ask for nothing (1) - int8-seeds: 3 criteria were declared after their evidence existed rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: all 17 record files are new, changed or deleted since commit 0e3550d9bdb6, which does not hold the record. Not committed: .rb/.gitattributes (new), .rb/.gitignore (new), .rb/experiments/int8-seeds/criteria/best-of-five.yaml (new) and 14 more To check it yourself, in your terminal: git -C ~/int8-seeds status --short --untracked-files=all -- .rb Rules four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58 reads: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb) result: measured: int8 on its own, with its interval, so neither better nor worse rule: recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4 timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds best-of-five INT8's best of 5 seeds reaches 91.65% top-1 recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65 reads: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb) result: point only: a best of N is one unit's own value, with no interval rule: recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65 timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds relative-drop INT8 loses at most 0.1% of FP32's top-1 estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1% reads: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: estimate meets criterion: estimate -0.0916% ≥ -0.1% timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds · the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval Experiments int8-seeds INT8 post-training quantization, five seeds · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 7 verified · 0 provisional · 0 unknown · 1 per-run · runs 5 to run again: code unknown · environment unknown · files ok Runs: 5 recorded on 1 experiment · every receipt is what rb wrote Next: rb confirm ``` - A criterion is a rule a result must meet. Its id, such as `four-of-five`, is its name in every command. - A count or best of N that is met reads **recorded values meet criterion**: the recorded seeds themselves meet it. - The relative change reads **estimate meets criterion**: the estimate meets the rule, but its interval crosses -0.1. ## What rb did, step by step 1. Declare the three rules ```rb-session frame="terminal" $ rb criterion add "INT8 reaches 91.58% top-1 on at least 4 of 5 seeds" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.58 --count 4 --of 5 --why docs/plan.md:4 --id four-of-five four-of-five: INT8 reaches 91.58% top-1 on at least 4 of 5 seeds 4 of 5 units' int8.top1 ≥ 91.58 · primary · on int8-seeds declared; no evidence recorded yet Next: rb run add int8-seeds --from --set seed= $ rb criterion add "INT8's best of 5 seeds reaches 91.65% top-1" -e int8-seeds -q q-seeds --metric int8.top1 --at-least 91.65 --best --of 5 --why docs/plan.md:5 --id best-of-five best-of-five: INT8's best of 5 seeds reaches 91.65% top-1 the best of 5 units' int8.top1 ≥ 91.65 · primary · on int8-seeds declared; no evidence recorded yet Next: rb run add int8-seeds --from --set seed= $ rb criterion add "INT8 loses at most 0.1% of FP32's top-1" -e int8-seeds -q q-seeds --metric change.top1 --at-least -0.1 --relative --why docs/plan.md:6 --id relative-drop relative-drop: INT8 loses at most 0.1% of FP32's top-1 change.top1 as a relative change ≥ -0.1% · primary · on int8-seeds declared; no evidence recorded yet Next: rb run add int8-seeds --from --set seed= ``` 2. Record each seed ```rb-session frame="terminal" $ rb run add int8-seeds --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml runbvd5mg on int8-seeds: fp32.top1=91.71, int8.top1=91.6 · seed=1 Ran with the spec: 5 of its settings found in results/int8-ptq/seed1/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) four-of-five criterion undetermined · point only: int8.top1: 91.6 % · mean, 1 seed (rb) best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 1 seed (rb) relative-drop criterion not met · point only: change.top1: -0.1199 % · relative change of the means, paired, 1 seed (rb) rb put the code in git at refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/d5832f1e0ac5298950edf64261220c331f21c020 removes it. Since this session last looked: - four-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1 (rb decide four-of-five accept|reject|investigate --why "...") - best-of-five: criterion undetermined: the rule is of 5 seeds; the runs hold 1 (rb decide best-of-five accept|reject|investigate --why "...") - relative-drop misses its rule and no call on it is recorded: INT8 loses at most 0.1% of FP32's top-1 (rb decide relative-drop accept|reject|investigate --why "...") - 4 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed2/, results/int8-ptq/seed3/, results/int8-ptq/seed4/ and 1 more (rb reconcile) Next: rb show four-of-five · rb show best-of-five · rb show relative-drop $ rb run add int8-seeds --from results/int8-ptq/seed2/metrics.json --set seed=2 --config results/int8-ptq/seed2/config.yaml run5zw7oj on int8-seeds: fp32.top1=91.66, int8.top1=91.58 · seed=2 Ran with the spec: 5 of its settings found in results/int8-ptq/seed2/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) four-of-five criterion undetermined · point only: int8.top1: 91.59 % · mean, 2 seeds (rb) best-of-five criterion undetermined · point only: int8.top1: 91.6 % · best unit seed 1, 2 seeds (rb) relative-drop criterion not met · point only: change.top1: -0.1036 % · relative change of the means, paired, 2 seeds (rb) Since this session last looked: - 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed3/, results/int8-ptq/seed4/, results/int8-ptq/seed5/ (rb reconcile) Next: rb show four-of-five · rb show best-of-five · rb show relative-drop $ rb run add int8-seeds --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml runongfds on int8-seeds: fp32.top1=91.74, int8.top1=91.69 · seed=3 Ran with the spec: 5 of its settings found in results/int8-ptq/seed3/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) four-of-five criterion undetermined · measured: int8.top1: 91.62 [91.48, 91.77] % · mean, t interval, 3 seeds, 95% (rb) best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 3 seeds (rb) relative-drop estimate meets criterion · loss: change.top1: -0.08724 [-0.1685, -0.005936] % · relative change of the means, paired, delta-method t interval, 3 seeds, 95% (rb) Since this session last looked: - 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/int8-ptq/seed4/, results/int8-ptq/seed5/ (rb reconcile) Next: rb show four-of-five · rb show best-of-five · rb show relative-drop $ rb run add int8-seeds --from results/int8-ptq/seed4/metrics.json --set seed=4 --config results/int8-ptq/seed4/config.yaml runmyetzp on int8-seeds: fp32.top1=91.69, int8.top1=91.57 · seed=4 Ran with the spec: 5 of its settings found in results/int8-ptq/seed4/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) four-of-five criterion undetermined · measured: int8.top1: 91.61 [91.52, 91.7] % · mean, t interval, 4 seeds, 95% (rb) best-of-five criterion undetermined · point only: int8.top1: 91.69 % · best unit seed 3, 4 seeds (rb) relative-drop estimate meets criterion · loss: change.top1: -0.09815 [-0.153, -0.04325] % · relative change of the means, paired, delta-method t interval, 4 seeds, 95% (rb) Since this session last looked: - results/int8-ptq/seed5/ holds results written before the record began, which no recorded experiment reads yet (rb reconcile) Next: rb show four-of-five · rb show best-of-five · rb show relative-drop $ rb run add int8-seeds --from results/int8-ptq/seed5/metrics.json --set seed=5 --config results/int8-ptq/seed5/config.yaml runovg3mt on int8-seeds: fp32.top1=91.72, int8.top1=91.66 · seed=5 Ran with the spec: 5 of its settings found in results/int8-ptq/seed5/config.yaml, all matching. Produced: 2026-10-20 10:00:00 (file) four-of-five recorded values meet criterion · measured: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb) best-of-five recorded values meet criterion · point only: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb) relative-drop estimate meets criterion · loss: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb) Next: rb show four-of-five · rb show best-of-five · rb show relative-drop ``` 3. Check the file format These criteria are saved in format 2. rb 0.7.0 and 0.7.1 write nothing to a record with a format-2 file, so upgrade everyone who shares it together. ```rb-session frame="terminal" $ head -3 .rb/experiments/int8-seeds/criteria/four-of-five.yaml rb: 2 kind: criterion id: four-of-five ``` 4. Read each criterion ```rb-session frame="terminal" $ rb show four-of-five Criterion four-of-five on int8-seeds (primary, own): INT8 reaches 91.58% top-1 on at least 4 of 5 seeds four-of-five INT8 reaches 91.58% top-1 on at least 4 of 5 seeds recorded values meet criterion · measured · 4 of 5 units' int8.top1 ≥ 91.58 reads: int8.top1: 91.62 [91.55, 91.69] % · mean, t interval, 5 seeds, 95% (rb) result: measured: int8 on its own, with its interval, so neither better nor worse rule: recorded values meet criterion: 4 of 5 seeds at least 91.58; the rule asks for 4 timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds integrity: ok runbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1 run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2 runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3 runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4 runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5 ``` ```rb-session frame="terminal" $ rb show best-of-five Criterion best-of-five on int8-seeds (primary, own): INT8's best of 5 seeds reaches 91.65% top-1 best-of-five INT8's best of 5 seeds reaches 91.65% top-1 recorded values meet criterion · point only · the best of 5 units' int8.top1 ≥ 91.65 reads: int8.top1: 91.69 % · best unit seed 3, 5 seeds (rb) result: point only: a best of N is one unit's own value, with no interval rule: recorded values meet criterion: best of 5 seeds, seed 3: 91.69 ≥ 91.65 timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds integrity: ok runbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1 run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2 runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3 runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4 runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5 ``` ```rb-session frame="terminal" $ rb show relative-drop Criterion relative-drop on int8-seeds (primary, own): INT8 loses at most 0.1% of FP32's top-1 relative-drop INT8 loses at most 0.1% of FP32's top-1 estimate meets criterion · loss · change.top1 as a relative change ≥ -0.1% reads: change.top1: -0.0916 [-0.1329, -0.05029] % · relative change of the means, paired, delta-method t interval, 5 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: estimate meets criterion: estimate -0.0916% ≥ -0.1% timing: declared after the evidence existed: all 5 units were produced before it was declared units: 5 seeds integrity: ok · the interval [-0.1329, -0.05029] crosses -0.1: met on the estimate, not across the whole interval runbvd5mg: int8.top1 91.6 (file) · unit seed=1 · produced before the declaration · seed=1 runbvd5mg: fp32.top1 91.71 (file) · unit seed=1 · produced before the declaration · seed=1 run5zw7oj: int8.top1 91.58 (file) · unit seed=2 · produced before the declaration · seed=2 run5zw7oj: fp32.top1 91.66 (file) · unit seed=2 · produced before the declaration · seed=2 runongfds: int8.top1 91.69 (file) · unit seed=3 · produced before the declaration · seed=3 runongfds: fp32.top1 91.74 (file) · unit seed=3 · produced before the declaration · seed=3 runmyetzp: int8.top1 91.57 (file) · unit seed=4 · produced before the declaration · seed=4 runmyetzp: fp32.top1 91.69 (file) · unit seed=4 · produced before the declaration · seed=4 runovg3mt: int8.top1 91.66 (file) · unit seed=5 · produced before the declaration · seed=5 runovg3mt: fp32.top1 91.72 (file) · unit seed=5 · produced before the declaration · seed=5 ``` --- # Share the record, and rerun it https://rabbitbrain.ai/docs/cookbooks/share-a-record/ A page lets your advisor read the work without your repository. A bundle lets their agent rerun it. ## Paste this into your agent **Write a page I can send, and a bundle anyone can rerun.** Give it the address you will host the folder at. ```text Share this Rabbit Brain record with my advisor. - Write one page I can send: `rb page`. - Write a bundle to host at https://example.org/int8-resnet/: `rb export --with-record --url https://example.org/int8-resnet/ --out ~/int8-site`. - Tell me what each one carries, as rb says it: the page alone, or the record too. ``` The example record: three INT8 ResNet-20 seeds run through `rb journal run`, with code in a git repository the reader can clone. ## What you'll see `rb page` writes one HTML file: ```rb-session frame="terminal" $ rb page rb-page.html written: the project as one file, readable anywhere; send it to anyone. It is the record as computed now; regenerate after new runs. If it ever disagrees with rb status, rb status is right. The page says: all record files committed · at commit 2d3d23f523f2. ``` `rb export --with-record --url` writes a folder for any static host: ```rb-session frame="terminal" $ rb export --with-record --url https://example.org/int8-resnet/ --out ~/int8-site ~/int8-site written: the whole record, the page, and a manifest hashing every file. Readable without rb; rb status inside record/ reopens it. Regenerate after new runs. The record keeps who made each write (human: or agent:, and the channel it came through), an agent's chat session, name, version, provider and model, the user's words passed with --words, and each run's host, platform and branch. Anyone who can read the bundle can read all of it. The page says: all record files committed · at commit 2d3d23f523f2. Code repository recorded: https://example.org/ada/int8-resnet.git at 2d3d23f523f2. Pass --no-repo to leave it out. Publish the directory at the address you gave; page.html is the reading surface and anyone reproduces it with: rb clone https://example.org/int8-resnet/bundle.zip ``` ```rb-session frame="terminal" $ ls ~/int8-site README.md bundle.zip manifest.json page.html record ``` `page.html`, open in a browser: Its **Rerun this research** link gives readers this prompt to paste into their own agent: **Rerun this research** The prompt on the page rb exported, word for word. ```text Rerun this research with Rabbit Brain (rb). Its record: https://example.org/int8-resnet/ 1. Install rb: pip install rabbit-brain 2. Clone the record into a new folder: rb clone https://example.org/int8-resnet/bundle.zip int8-resnet-20-on-cifar-10. This bundle names the code: rb clone gets it too, at the commit the record names, and puts the record in it. If it says it could not get the code, tell me what it says and stop: once I give you an address of the code, rb clone https://example.org/int8-resnet/bundle.zip --code gets it. 3. In that folder, run rb context and read it before anything else. Its "Rerun the parent" part names the code and whether it is here, each resource, such as a model or a dataset, with how to get it, and the first command for each experiment. 4. Get those resources. Run each experiment's commands in the order rb context names them, each as rb journal run -- ; where rb context names a command that puts a run's code here, run that one first. After each, run rb reconcile and record the run it proposes with the rb run add line it prints. 5. Then run rb status, and tell me which rules reproduced and which did not, with their numbers. Do not change a rule. If something cannot run here, say what and why. ``` ## What the reader's agent did We pasted that prompt into a fresh agent session in another folder, with the bundle saved as `int8-record.zip`. 1. Clone the record ```rb-session frame="terminal" $ rb clone int8-record.zip int8-advisor int8-advisor: "INT8 ResNet-20 on CIFAR-10" cloned: 1 experiment, 1 criterion, 1 metric, 0 resources. Everything arrived inherited; nothing counts until it is reproduced here. It is a checkout of the code at 2d3d23f523f2, on branch rerun; the record is its latest commit, in place of the parent's, which git keeps at 2d3d23f523f2. The parent's runs, analyses and decisions stay in the parent's record. Record runs made in this repository, and each inherited criterion reads them like any finding: reproduced or not reproduced. The code: https://example.org/ada/int8-resnet.git at 2d3d23f523f2, checked out here. The parent's record is taken out of this branch; git keeps it at that commit, and the record here is this line of work's own. Next: cd int8-advisor && rb context (the parent, the code, each resource and the first command to rerun) · rb env check (does this machine match the runs that made it?) · rb export --with-record --url (publish your fork) ``` ```rb-session frame="terminal" $ git log --oneline -n 3 e370522 rerun of int8-resnet-20-on-cifar-10 from int8-record.zip: its record, inherited 2d3d23f Record the INT8 runs 12ec61a Ignore results ``` 2. Read what it takes to rerun ```rb-session frame="terminal" $ rb context # The record: INT8 ResNet-20 on CIFAR-10 Generated by `rb context` from `.rb/`; read the state from here or `rb status --json`, never from memory or a summary. ## Rerun the parent - parent: INT8 ResNet-20 on CIFAR-10 (cloned 2026-11-10 from int8-record.zip) - code: https://example.org/ada/int8-resnet.git at 2d3d23f523f2, here: your checkout's code is commit 2d3d23f523f2's, the record aside - int8-ptq, first, for top1-drop-budget: `rb journal run -- python eval.py --seed 1` (the parent's run runzmsx6g of fp32 and int8, its command as given; per run seed=1; ran at 12ec61a726fe; your checkout's code is commit 12ec61a726fe's); no run recorded here yet … (97 more lines) ``` `top1-drop-budget` is the id of the record's criterion, the rule its runs are checked against. 3. Run each command, and record what rb proposes ```rb-session frame="terminal" $ rb journal run -- python eval.py --seed 1 {"metrics": {"fp32.top1": 91.71, "int8.top1": 91.6}} exit 0. The command, its exit code and what it wrote are in the journal. It printed its metrics: fp32.top1=91.71, int8.top1=91.6. rb reconcile proposes the run of int8-ptq that records them. Since this session last looked: - results/int8-ptq/seed1/ holds results no recorded experiment reads yet (rb reconcile) - results/int8-ptq/seed1/metrics.json: results landed and are not in the record (rb reconcile) Next: rb reconcile ``` ```rb-session frame="terminal" $ rb reconcile The repository at commit e3705223a918: 1 results folder. results/int8-ptq/seed1/ holds results files no recorded run named. With no report of its own it is no experiment: rb run add --from records a run from it. Results in the repository, not in the record (1): results/int8-ptq/seed1/metrics.json: fp32.top1=91.71, int8.top1=91.6 rb run add int8-ptq --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml --commit e3705223a918d0342d715d3f0a3b9e325f0a3394 --command 'python eval.py --seed 1' rb reconcile --apply keeps this in the record: 1 run to record. Next: rb reconcile --apply · rb run add int8-ptq --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml --commit e3705223a918d0342d715d3f0a3b9e325f0a3394 --command 'python eval.py --seed 1' ``` ```rb-session frame="terminal" $ rb run add int8-ptq --from results/int8-ptq/seed1/metrics.json --set seed=1 --config results/int8-ptq/seed1/config.yaml --commit e3705223a918d0342d715d3f0a3b9e325f0a3394 --command "python eval.py --seed 1" runjjmuh4 on int8-ptq: fp32.top1=91.71, int8.top1=91.6 · seed=1 Ran with the spec: 5 of its settings found in results/int8-ptq/seed1/config.yaml, all matching. Produced: 2026-11-10 15:32:30 (file) top1-drop-budget criterion undetermined · point only: change.top1: -0.11 % · paired difference, 1 seed (rb) rb put the code in git at refs/rb/snapshots/f42c461f7691b56287e29a7e9954bc25f90869c1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/f42c461f7691b56287e29a7e9954bc25f90869c1 removes it. Next: rb show top1-drop-budget ``` After the third seed: ```rb-session frame="terminal" $ rb run add int8-ptq --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml --commit e3705223a918d0342d715d3f0a3b9e325f0a3394 --command "python eval.py --seed 3" run47yneq on int8-ptq: fp32.top1=91.74, int8.top1=91.69 · seed=3 Ran with the spec: 5 of its settings found in results/int8-ptq/seed3/config.yaml, all matching. Produced: 2026-11-10 15:32:32 (file) top1-drop-budget reproduced · loss: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show top1-drop-budget ``` 4. Read what the criteria became ```rb-session frame="terminal" $ rb status INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10 4 record files not committed Rerun the parent - parent: INT8 ResNet-20 on CIFAR-10 (cloned 2026-11-10 from int8-record.zip) - code: https://example.org/ada/int8-resnet.git at 2d3d23f523f2, here: your checkout's code is commit 2d3d23f523f2's, the record aside - int8-ptq, first, for top1-drop-budget: `rb journal run -- python eval.py --seed 1` (the parent's run runzmsx6g of fp32 and int8, its command as given; per run seed=1; ran at 12ec61a726fe; your checkout's code is commit 12ec61a726fe's); 3 runs recorded here Does INT8 lose at most 0.5 points of top-1 against FP32 on CIFAR-10? (open) INT8 loses at most 0.5 points of top-1 against FP32. change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb); loss: the interval clear of 0 on the worse side (higher is better); reproduced: estimate -0.08 ≥ -0.5; rule and question link declared before the evidence (by production times). units: 3 seeds; optim.lr = 0.1 is inherited from the parent project; optim.momentum = 0.9 is inherited from the parent project; data.batch_size = 128 is inherited from the parent project; eval.batch_size = 256 is inherited from the parent project; fp32.precision = fp32 is inherited from the parent project; int8.precision = int8 is inherited from the parent project; int8.calib_images = 512 is inherited from the parent project. Rules: 1 reproduced The design of 1 experiment is not confirmed: rb confirm lists it. rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-10 15:32 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run. rb checked the record with git at 2026-11-10 15:32 UTC: 4 of its 13 record files are new, changed or deleted since commit e3705223a918. Not committed: .rb/log.jsonl (changed), .rb/experiments/int8-ptq/runs/run47yneq.yaml (new), .rb/experiments/int8-ptq/runs/runjjmuh4.yaml (new) and 1 more To check it yourself, in your terminal: git -C ~/advisor/int8-advisor status --short --untracked-files=all -- .rb Rules top1-drop-budget INT8 loses at most 0.5 points of top-1 against FP32 reproduced · loss · change.top1 ≥ -0.5 (n ≥ 3) reads: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: reproduced: estimate -0.08 ≥ -0.5 timing: declared before the evidence (by production times) units: 3 seeds · optim.lr = 0.1 is inherited from the parent project · optim.momentum = 0.9 is inherited from the parent project · data.batch_size = 128 is inherited from the parent project and 4 more: rb show top1-drop-budget Experiments int8-ptq INT8 post-training quantization · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 0 verified · 0 provisional · 0 unknown · 7 inherited · 1 per-run · runs 3 to run again: code ok · environment ok · files ok Runs: 3 recorded on 1 experiment · every receipt is what rb wrote Nothing open. ``` rb 0.7.0 and 0.7.1 may refuse to clone a bundle made with rb 0.8.4. Upgrade everyone who shares a record together. ## On rabbitbrain.ai [rabbitbrain.ai](https://rabbitbrain.ai) serves shared record pages, and samples at [rabbitbrain.ai/demo](https://rabbitbrain.ai/demo), each with the same **Rerun this research** prompt. `rb login` and `rb sync` put a private copy of your record's page in a rabbitbrain.ai account. Anyone can sign up at [rabbitbrain.ai/signin](https://rabbitbrain.ai/signin). With `rb sync --with-record`, a share link you make from that page lets readers rerun the record until you revoke the link. ```rb-session frame="terminal" $ rb login First, this terminal gets a code: NN6T-D43Q Open https://rabbitbrain.ai/link?code=NN6T-D43Q on any device; it signs you in first if needed, and approve the code. ``` --- # Rerun a run from its receipt https://rabbitbrain.ai/docs/cookbooks/rerun-from-receipt/ A run's receipt names its command and the commit it ran on. Rerun the command on that commit and record the rerun as a check. ## Paste this into your agent **Rerun seed 2 from its receipt, on the code it ran on, and check it.** ```text Rerun seed 2 of int8-ptq from its Rabbit Brain receipt, and tell me if it still gives the same numbers. - Run `rb context --model ` first. - Read the run's receipt with `rb show `: the command, and the commit it ran on. - Check out that commit in a worktree inside this repository (`git worktree add .worktrees/rerun `), and run the receipt's command there through `rb journal run`. - Record the new run as a check of the old one: `--repeats --again --why "rerun from its receipt"`, with `--commit`. Finish with `rb show` of the criterion exactly as printed, and what rb says of the repeat. ``` Since the runs, one commit changed how `eval.py` builds its output path. ## What you'll see ```rb-session frame="terminal" $ rb show top1-drop-budget Criterion top1-drop-budget on int8-ptq (primary, own): INT8 loses at most 0.5 points of top-1 against FP32 top1-drop-budget INT8 loses at most 0.5 points of top-1 against FP32 criterion met · loss · change.top1 ≥ -0.5 (n ≥ 3) reads: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: criterion met: estimate -0.08 ≥ -0.5 timing: declared before the evidence (by production times) units: 3 seeds from 4 runs, 1 repeat integrity: ok · runt6sqax repeats runssvuju: the same number, a check that holds rung2qpud: difference -0.11 (file) · unit seed=1 · seed=1 runssvuju: difference -0.08 (file) · unit seed=2 · seed=2 runqllqdf: difference -0.05 (file) · unit seed=3 · seed=3 runt6sqax: difference -0.08 (file) · unit seed=2 · repeats runssvuju · seed=2 ``` The rerun is a repeat of seed 2, so the criterion, a rule a result must meet, still rests on 3 seeds: "from 4 runs, 1 repeat". ## What rb did, step by step 1. See what a rerun needs ```rb-session frame="terminal" $ rb show int8-ptq Experiment int8-ptq: INT8 post-training quantization · open · under q-int8 design not confirmed · list it to confirm: rb confirm int8-ptq hypothesis or purpose: not recorded why: not recorded variants: fp32 (baseline) · int8 (candidate) varies: precision, calib_images spec 85201d31eae8 · frozen 2026-11-10T15:32:35.679+00:00 (85201d31eae8) · note: before the runs · not confirmed settings: optim.lr 0.1 verified · configs/train.yaml#optim.lr optim.momentum 0.9 verified · configs/train.yaml#optim.momentum data.batch_size 128 verified · configs/train.yaml#data.batch_size eval.batch_size 256 verified · configs/train.yaml#eval.batch_size seed (each run) per-run · no source fp32.precision fp32 verified · configs/fp32.yaml#precision int8.precision int8 verified · configs/int8.yaml#precision int8.calib_images 512 verified · configs/int8.yaml#calib_images criteria: top1-drop-budget INT8 loses at most 0.5 points of top-1 against FP32 criterion met · loss · change.top1 ≥ -0.5 (n ≥ 3) reads: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) result: loss: the interval clear of 0 on the worse side (higher is better) rule: criterion met: estimate -0.08 ≥ -0.5 timing: declared before the evidence (by production times) units: 3 seeds integrity: ok runs: rung2qpud, runssvuju, runqllqdf to run it again: code ok commit 12ec61a726fe code · commit 12ec61a726fe, which all 3 runs ran at your checkout differs in 1 file: eval.py show code diff: git diff 12ec61a726feda8c90dbba424b07ee7ec0b9d32c -- ":(top)" ":(top,exclude).rb" rebuild: git worktree add --detach ../int8-ptq-12ec61a 12ec61a726feda8c90dbba424b07ee7ec0b9d32c environment ok python 3.14.4 files ok all 3 files the runs named are here unchanged ``` 2. Read the run's receipt ```rb-session frame="terminal" $ rb show runssvuju Run runssvuju on int8-ptq (file) produced 2026-11-10 15:32 UTC (its file's time) · recorded 2026-11-10 15:32 UTC fp32.top1 = 91.66 int8.top1 = 91.58 per-run: seed=2 command (as given): python eval.py --seed 2 (rb journal run watched it make these numbers, 2026-11-10) code that made it: 12ec61a726fe (as given) your checkout differs in 1 file: eval.py repository when recorded: commit 12ec61a726fe config results/int8-ptq/seed2/config.yaml: 5 match, 0 differ, 0 absent file results/int8-ptq/seed2/metrics.json sha256 342131bfa92f (72 bytes) · here ``` 3. See what changed since ```rb-session frame="terminal" $ git diff 12ec61a -- eval.py diff --git a/eval.py b/eval.py index 35a1cb3..901a798 100644 --- a/eval.py +++ b/eval.py @@ -7,7 +7,7 @@ p = argparse.ArgumentParser() p.add_argument("--seed", type=int, required=True) a = p.parse_args() fp32, int8 = TOP1[a.seed] -out = f"results/int8-ptq/seed{a.seed}" +out = os.path.join("results", "int8-ptq", f"seed{a.seed}") os.makedirs(out, exist_ok=True) with open(f"{out}/metrics.json", "w") as f: json.dump({"fp32": {"top1": fp32}, "int8": {"top1": int8}}, f, indent=2) ``` 4. Check out the receipt's commit in a worktree ```rb-session frame="terminal" $ git worktree add -q .worktrees/rerun 12ec61a ``` Then, in `.worktrees/rerun`: ```rb-session frame="terminal" $ rb context # The record: INT8 ResNet-20 on CIFAR-10 This worktree (no branch · ~/int8-rerun/.worktrees/rerun) has no record of its own: rb writes to the record in main · ~/int8-rerun, and each run keeps this worktree's code. … (98 more lines) ``` 5. Run the receipt's command ```rb-session frame="terminal" $ rb journal run -- python eval.py --seed 2 {"metrics": {"fp32.top1": 91.66, "int8.top1": 91.58}} exit 0. The command, its exit code and what it wrote are in the journal. It printed its metrics: fp32.top1=91.66, int8.top1=91.58. rb reconcile proposes no new run: a recorded run of int8-ptq holds these numbers. Next: rb reconcile ``` 6. Record it as a check The output matches the first run's byte for byte. `--again --why` records it anyway, as a second execution, and `--repeats` makes it a check of seed 2, not a new seed. ```rb-session frame="terminal" $ rb run add int8-ptq --from results/int8-ptq/seed2/metrics.json --set seed=2 --config results/int8-ptq/seed2/config.yaml --command "python eval.py --seed 2" --commit 12ec61a726feda8c90dbba424b07ee7ec0b9d32c --repeats runssvuju --again --why "rerun from its receipt" runt6sqax on int8-ptq: fp32.top1=91.66, int8.top1=91.58 · seed=2 Ran with the spec: 5 of its settings found in .worktrees/rerun/results/int8-ptq/seed2/config.yaml, all matching. It repeats runssvuju: a check of that run, in its unit, never a new one. Produced: 2026-11-10 15:32:38 (file) top1-drop-budget criterion met · loss: change.top1: -0.08 [-0.1545, -0.005476] % · paired difference, t interval, 3 seeds, 95% (rb) Next: rb show top1-drop-budget ``` ```rb-session frame="terminal" $ rb show runt6sqax Run runt6sqax on int8-ptq (file) produced 2026-11-10 15:32 UTC (its file's time) · recorded 2026-11-10 15:32 UTC fp32.top1 = 91.66 int8.top1 = 91.58 per-run: seed=2 command (as given): python eval.py --seed 2 (rb journal run watched it make these numbers, 2026-11-10) code that made it: 12ec61a726fe (as given) your checkout has this code repository when recorded: commit 12ec61a726fe config .worktrees/rerun/results/int8-ptq/seed2/config.yaml: 5 match, 0 differ, 0 absent file .worktrees/rerun/results/int8-ptq/seed2/metrics.json sha256 342131bfa92f (72 bytes) · here note: recorded again: rerun from its receipt. · not confirmed ``` --- # rb with Claude Code, Codex and Cursor https://rabbitbrain.ai/docs/cookbooks/three-agents/ Each agent needs rb's instructions where it reads them, so all three write to the same record. ## Paste this into your agent **Set this repository up for every agent you use, and say what goes in git.** ```text Set this repository up so Claude Code, Codex and Cursor all keep the same Rabbit Brain record. - A session that is already running loads no new skill, so `rb init` installs nothing in this session. Tell me to run `rb init --dry-run`, then `rb init ""`, in my own terminal. - After I have, run `git status --short`, and tell me which files rb added for each agent and which ones to commit. ``` The repository already has a `CLAUDE.md`, an `AGENTS.md` of its own, and a `.cursor/` folder. ## What you'll see `rb init`, in your own terminal: ```rb-session frame="terminal" $ rb init "INT8 ResNet-20 on CIFAR-10" Project 'int8-resnet-20-on-cifar-10' started: ~/int8-agents/.rb Rabbit Brain YAML files beside the code: commit .rb/ with it, so the record travels with the work. For the agents here: Claude Code (CLAUDE.md): the skill in .claude/skills/rabbit-brain/SKILL.md; rb's MCP server in a new .mcp.json; rb's session hook in a new .claude/settings.json (SessionStart and Stop run rb journal scan, which looks at the repository and says nothing) The agents that read AGENTS.md (AGENTS.md): AGENTS.md was here before rb, so rb left it as it is. rb skill add --agents-md --yes adds rb's section at its end (the text rb skill prints, between two marker lines). Cursor (.cursor/): .cursor/rules/rabbit-brain.mdc; rb's session hook in a new .cursor/hooks.json (sessionStart and stop run rb journal scan, which looks at the repository and says nothing) Then paste this line into your agent's chat: Use Rabbit Brain to track this: read rb context, then record where the work stands. ``` rb asks once before it changes a file that was already there. We ran it where it could not ask, so it left `AGENTS.md` and named the command that adds rb's section: ```rb-session frame="terminal" $ rb skill add --agents-md --yes wrote ~/int8-agents/AGENTS.md The agent picks it up in its next session. The skill defers to `rb docs` for the full rules. ``` ```rb-session frame="terminal" $ git status --short M AGENTS.md ?? .claude/ ?? .cursor/ ?? .mcp.json ?? .rb/ ``` | Agent | What rb wrote | Where | | --- | --- | --- | | Claude Code | the skill, rb's MCP server, and rb's session hook | `.claude/skills/rabbit-brain/SKILL.md`, `.mcp.json`, `.claude/settings.json` | | Codex, and any agent that reads `AGENTS.md` | rb's section, between two marker lines, at the end of the file | `AGENTS.md` | | Cursor | a rule, and rb's session hook | `.cursor/rules/rabbit-brain.mdc`, `.cursor/hooks.json` | There is no `.codex/` here, so rb wrote no hook for Codex. With `.codex/`, or when Codex runs `rb init`, rb adds one to `.codex/hooks.json`. [The session hook](https://rabbitbrain.ai/docs/connect/#the-session-hook) says what it does. Commit all of it with `.rb/`. ## What rb did, step by step 1. Show the plan, write nothing In your own terminal: ```rb-session frame="terminal" $ rb init --dry-run Dry run: nothing was written. The write would say: Project 'int8-agents' started: ~/int8-agents/.rb Rabbit Brain YAML files beside the code: commit .rb/ with it, so the record travels with the work. For the agents here: Claude Code (CLAUDE.md): the skill in .claude/skills/rabbit-brain/SKILL.md; rb's MCP server in a new .mcp.json; rb's session hook in a new .claude/settings.json (SessionStart and Stop run rb journal scan, which looks at the repository and says nothing) The agents that read AGENTS.md (AGENTS.md): AGENTS.md was here before rb, so rb left it as it is. rb skill add --agents-md --yes adds rb's section at its end (the text rb skill prints, between two marker lines). Cursor (.cursor/): .cursor/rules/rabbit-brain.mdc; rb's session hook in a new .cursor/hooks.json (sessionStart and stop run rb journal scan, which looks at the repository and says nothing) Then paste this line into your agent's chat: Use Rabbit Brain to track this: read rb context, then record where the work stands. rb init would write: .rb/: the record (.gitignore, .gitattributes, project.yaml, log.jsonl), and rb's lock and local copies (.lock, cache/), which .rb/.gitignore keeps out of git .claude/skills/rabbit-brain/SKILL.md: new .mcp.json: new .claude/settings.json: new .cursor/rules/rabbit-brain.mdc: new .cursor/hooks.json: new .git/rb/owned.jsonl: rb's list of what it writes outside .rb/ .git/rb/journal/: what rb sees land in the repository, from the look rb init makes as the record begins Later commands add: refs/rb/snapshots/: a ref in git to the code a run was recorded at; git update-ref -d removes it (rb run add, rb journal run) .git/rb/views/: a past version of the record, checked out while the screen shows it (rb open, rb ui) .git/rb-last-seen.json: what you last saw, so rb ui can say what is new since (rb ui) rb-page.html: the record as one page, beside .rb/ by default (rb page) rb-export/: the page with a manifest, beside .rb/ by default (rb export) rb-export.zip: the same, as one zip file (rb export --zip) ``` 2. In an agent session, install nothing ```rb-session frame="terminal" $ rb init "INT8 ResNet-20 on CIFAR-10" Project 'int8-resnet-20-on-cifar-10' started: ~/int8-agents-session/.rb Rabbit Brain YAML files beside the code: commit .rb/ with it, so the record travels with the work. For the agents here: Nothing installed: a session already running loads no skill or MCP server added during it. For later sessions: Claude Code: rb skill add --claude --project, and claude mcp add --scope project rabbit-brain -- rb mcp The agents that read AGENTS.md: rb skill add --agents-md Cursor: rb skill add --cursor Use Rabbit Brain to track this: read rb context, then record where the work stands. Next: rb context --model ``` 3. rb's MCP server, for Claude Code ```rb-session frame="terminal" $ cat .mcp.json { "mcpServers": { "rabbit-brain": { "command": "rb", "args": [ "mcp" ] } } } ``` `claude mcp add --scope project rabbit-brain -- rb mcp` adds the same entry by hand. 4. rb's session hook, for Cursor ```rb-session frame="terminal" $ cat .cursor/hooks.json { "version": 1, "hooks": { "sessionStart": [ { "command": "rb journal scan --hook cursor; exit 0", "timeout": 10 } ], "stop": [ { "command": "rb journal scan --hook cursor; exit 0", "timeout": 10 } ] } } ``` 5. rb's section, for Codex ```rb-session frame="terminal" $ tail -n 12 AGENTS.md - work in another branch or worktree is recorded there the same way, never as an experiment per branch: name the experiment and the variant on every run, and the records merge with git (`rb status` says what branches not merged yet hold). ## Boundaries - Never compute a finding, an error, a ranking or a regression yourself: record the numbers, and read the finding. - Never edit `.rb/` by hand: rb catches it, and `rb doctor` lists it. - Never set or unset `RB_ACTOR` or `RB_HUB_TOKEN`, and never read the credentials file `rb login` writes. - Report a finding by its axes as rb prints them: the comparison with its interval and units, the rule, the timing, and every integrity problem. `criterion undetermined · point only` on 2 units is said in those words, never rounded up to "it works". Everything else is in `rb docs`. Tools instead of shell: `claude mcp add rabbit-brain -- rb mcp` from the project root; same record, same rules. ``` 6. Take it all out `rb remove` takes out only what is rb's: ```rb-session frame="terminal" $ rb remove --dry-run Dry run: nothing was removed. rb remove would do this in ~/int8-agents: Will remove: .rb/: the record (1 record, each as its log says rb wrote it, and the log), and rb's own files and local copies .claude/skills/rabbit-brain/SKILL.md: rb's skill for Claude Code, as rb wrote it .cursor/rules/rabbit-brain.mdc: rb's rule for Cursor, as rb wrote it AGENTS.md: rb's section, as rb added it: the file goes back to what it was, byte for byte .mcp.json: made by rb, and as rb wrote it .claude/settings.json: made by rb, and as rb wrote it .cursor/hooks.json: made by rb, and as rb wrote it .git/rb/: rb's folder inside git's: its journal, its list of what it wrote outside .rb/, and the past versions it showed Not rb's to touch: your research files, worktrees and .git/worktrees, git history, git objects, config, hooks and the index. Listed, not removed: the rb command itself: pip uninstall rabbit-brain (or uv tool uninstall rabbit-brain, pipx uninstall rabbit-brain) 4 record files not committed. To keep a copy first, run: rb export --with-record --zip --out ~/int8-resnet-20-on-cifar-10-record.zip ``` ## Who wrote what rb records which agent made each write, with its chat session and the model it reported (`rb log --json`). The screens never name the agent. --- # The CLI https://rabbitbrain.ai/docs/reference/cli/ Generated from the `--help` of rabbit-brain 0.8.4. With `--json`, a command prints exactly one JSON object on standard output, the envelope: `ok`, `rb_version`, `command`, `data`, `errors`, `next` and `rb_delta`. `rb schema envelope` prints its shape. | Exit code | What it means | | --- | --- | | `0` | ok | | `1` | a check that ran and failed: a gate of `rb status --fail-on`, a setting that did not verify, a problem `rb doctor` found, a file `rb fmt --check` would rewrite, a file `rb show` read that is not a valid record, or the command `rb journal run` wrapped. The JSON has `ok: true`, because the result is data. | | `2` | rb could not do what was asked, and says why | | `3` | a fault in rb itself | | Group | Commands | | --- | --- | | [The record](https://rabbitbrain.ai/docs/reference/cli/the-record/) | `init`, `project`, `question`, `experiment`, `variant`, `note`, `metric`, `spec`, `criterion`, `run`, `analysis`, `resource` | | [Calls](https://rabbitbrain.ai/docs/reference/cli/calls/) | `freeze`, `decide`, `confirm`, `retract` | | [Reading it](https://rabbitbrain.ai/docs/reference/cli/reading/) | `open`, `ui`, `status`, `show`, `compare`, `log`, `context`, `page` | | [Carrying it](https://rabbitbrain.ai/docs/reference/cli/carrying/) | `export`, `clone`, `env` | | [For agents](https://rabbitbrain.ai/docs/reference/cli/for-agents/) | `mcp`, `skill`, `journal`, `reconcile`, `compile` | | [Upkeep](https://rabbitbrain.ai/docs/reference/cli/upkeep/) | `doctor`, `fmt`, `telemetry`, `config` | | [Leaving](https://rabbitbrain.ai/docs/reference/cli/leaving/) | `remove` | | [Reference](https://rabbitbrain.ai/docs/reference/cli/reference/) | `tour`, `docs`, `schema`, `version`, `feedback` | | [rabbitbrain.ai](https://rabbitbrain.ai/docs/reference/cli/rabbitbrain-ai/) | `login`, `logout`, `sync` | ## rb --help ```text usage: rb [...] The record of the experiments your coding agents run, as Rabbit Brain YAML files in your repo: questions, experiments with their hypothesis, variants and criteria, runs with receipts, and the resources they used. Agents record; rb checks and computes findings; you decide. Documentation: rb docs. New here? Run rb tour. The record: init start the record (.rb/) here project title: the project's title question add|move|reword: what the work is trying to find out, as a tree experiments are filed under experiment add|file|relate|gate: what is tested: its purpose, hypothesis, variants and settings; where it is filed; how it stands to others; the decision it gates variant add: an arm of an experiment: baseline, candidate, control, ablation note add: words about an experiment or the project: an assumption, a caveat, context; a note changes no finding metric add: the catalogue: a metric's unit, which way is better, its other names spec set|verify|vary: an experiment's settings, where each value comes from, and whether rb has checked it criterion add|file|relate: a rule over one comparison, declared before the evidence it reads run add: what ran: its results, per-run settings, inputs and outputs, with a receipt analysis add: a computation over saved outputs, and the comparisons it produced with their intervals resource add|locate|expose: a dataset, checkpoint, code or artifact: what it is, where its bytes are, how to get it again, who has looked at it Calls: freeze lock an experiment's spec and criteria before the evidence, recorded with who locked it decide a call on a criterion, an experiment, a question or a setting: accept, reject, investigate; continue, park, stop, pivot, ship. Recorded with who made it and why; in an agent session it is proposed until the user confirms it confirm make a call proposed, or a past call mentioned, in an agent session yours, or an experiment's design one wrote: run in your own terminal; with no id, list what waits retract take anything back: it stays on record, stops counting, and rb says what rested on it Reading it: open the catalogue in your browser, live and read-only, on 127.0.0.1 only ui open this project in your terminal: your call, the cards, the tree (bare rb does this too) status each criterion on its axes, and what is open, with who usually acts show one record: a criterion with its finding, an experiment with its spec, a run with its receipt, e1/lr with its history; or a .yaml record file as canonical JSON compare an experiment's own table: variants by metrics, with changes against the baseline log every write, newest last: who, how, what context the record as one Markdown briefing a fresh agent session reads first page write the project as one self-contained, read-only HTML file you can send anywhere Carrying it: export write the project's page with a hashed manifest, readable without rb; with --with-record, the record too, as a portable bundle clone start your own line of work from an exported bundle: the structure arrives inherited, and nothing counts until reproduced here env the environment: check this machine against the record's runs and any env.* settings For agents: mcp serve the record to an agent over MCP (stdio); every call is recorded as the agent's skill print the agent skill; `rb skill add` installs it for your coding agent journal what rb observed in the repository: files that appeared, git moves, wrapped commands reconcile compile the repository against the record: each results folder with a report as an experiment, what its files say (verbatim), its checks, how to reproduce it; --apply keeps it compile list what existing reports and results files the record can place, and what it cannot; writes nothing Upkeep: doctor check the record: who you are recorded as, edits made outside rb, merge leftovers, sources outside the repository fmt write Rabbit Brain YAML files in their canonical form, comments kept (default: every record file under .rb/) telemetry optional usage analytics: off until you enable it; never sends research content config rb's own settings on this machine: whether rb keeps itself up to date Leaving: remove take rb out of this checkout: only what checks out as rb's (the record, its instructions for agents, its pages and exports, its refs and folders in git), after showing the plan Reference: tour a one-minute walkthrough: builds a small real project in ./rb-tour and runs the whole loop docs print AGENTS.md (--start: how to start the record in a project already under way; --errors: the error table) schema print a JSON Schema: `rb schema all` is Rabbit Brain formats 1 and 2 (rb-v2.schema.json), `rb schema ` one record's shape version print the version feedback a link to send feedback on rb: the page shows what it would send before you send it; rb sends nothing rabbitbrain.ai: login rabbitbrain.ai, optional: sign this machine in with a short code, approved on any device where you are signed in logout revoke this machine's hub credential and remove it from the credentials file sync rabbitbrain.ai, optional: a private copy of this record's page in your account, for your own eyes from anywhere rb --help for its arguments. Start: rb init "", then rb question add for each question the work asks, and rb context. ``` --- # Commands: The record https://rabbitbrain.ai/docs/reference/cli/the-record/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb init Starts the record (`.rb/`) in the current directory. ```text usage: rb init [-h] [--json] [--id ID] [--no-agents] [--new-project] [--yes] [--dry-run] [title] positional arguments: title the work's theme, which its questions share: a paper's title, a model, a line of work (default: this directory's name) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --id ID a short id for the project (default: from the title) --no-agents leave the agents' instruction files alone (default: install rb's instructions for the agents found here, as rb skill add does) --new-project start a separate project in a worktree whose repository already holds the project's record in another worktree (default: refused, with how to bring that record here) --yes add rb's section to an AGENTS.md and rb's server to a .mcp.json that are here without asking (default: asked once in your terminal; elsewhere left as they are, with what to add) --dry-run list every file and integration rb init would write, every file it would change and how, and what later commands add, writing nothing ``` ## rb project ```text usage: rb project [-h] ... positional arguments: <title> title change the project's title, keeping its identity and record options: -h, --help show this help message and exit ``` ### rb project title ```text usage: rb project title [-h] [--json] [--dry-run] -m WHY [--words TEXT] title positional arguments: title the work's theme: a paper's title, a model, a line of work options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would -m WHY, --why WHY why the title is changing, kept in the log --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb question ```text usage: rb question [-h] <add|move|reword> ... positional arguments: <add|move|reword> add add a question, at the top or under another move put a question under another one, or at the top; its evidence goes with it reword give a question new wording, kept as a revision: the log keeps the old wording, who changed it and why; its id, evidence and calls stay options: -h, --help show this help message and exit ``` ### rb question add ```text usage: rb question add [-h] [--json] [--dry-run] [--id ID] [--parent PARENT] [--words TEXT] text positional arguments: text options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --id ID --parent PARENT the question this one sits under --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ### rb question move ```text usage: rb question move [-h] [--json] [--dry-run] [--parent PARENT] [--top] [-m WHY] [--words TEXT] question positional arguments: question options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --parent PARENT --top -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ### rb question reword ```text usage: rb question reword [-h] [--json] [--dry-run] -m WHY [--words TEXT] question text positional arguments: question text the new wording, verbatim: the user's own words, never a formalization of them options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would -m WHY, --why WHY why it is reworded, kept in the log --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb experiment ```text usage: rb experiment [-h] <add|file|relate|gate> ... positional arguments: <add|file|relate|gate> add add an experiment: planned until its first run file file an experiment under one more question (or --remove it from one) relate how an experiment stands to another: reproduces, extends, follows_from, compares_with gate the decision an experiment gates, set or revised: the action taken if it holds, and if not; the sentence it was said in goes in --words options: -h, --help show this help message and exit ``` ### rb experiment add ```text usage: rb experiment add [-h] [--json] [--dry-run] [--id ID] [-q QUESTION] [--hypothesis HYPOTHESIS] [-m WHY] [--purpose PURPOSE] [--baseline BASELINE] [--candidate CANDIDATE] [--varies VARIES] [--like LIKE] [--note NOTE] [--if-holds TEXT] [--if-not TEXT] [--words TEXT] title positional arguments: title options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --id ID a short id you choose (default: e and six random characters) -q QUESTION, --question QUESTION a question it is filed under (repeatable: an experiment can answer several) --hypothesis HYPOTHESIS what you expect it to show -m WHY, --why WHY why you expect it --purpose PURPOSE what it is for, when that is not a hypothesis (a benchmark, a sweep) --baseline BASELINE name of the baseline variant, e.g. fp32 --candidate CANDIDATE name of a candidate variant, e.g. int8 (repeatable) --varies VARIES the settings the variants differ in on purpose (repeatable or comma-separated); any other difference is a confound --like LIKE copy another experiment's shared settings (as provisional) and what it varies --note NOTE a first note: context, an assumption, a caveat --if-holds TEXT the decision it gates: the action taken when it holds; the sentence it was said in goes in --words --if-not TEXT and what to do when it does not --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ### rb experiment file ```text usage: rb experiment file [-h] [--json] [--dry-run] [--remove] experiment question positional arguments: experiment question options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --remove ``` ### rb experiment relate ```text usage: rb experiment relate [-h] [--json] [--dry-run] [--note NOTE] experiment {reproduces,extends,follows_from,compares_with} target positional arguments: experiment {reproduces,extends,follows_from,compares_with} target another experiment's id here, its global uid, or a published manifest's address options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --note NOTE ``` ### rb experiment gate ```text usage: rb experiment gate [-h] [--json] [--dry-run] [--if-holds TEXT] [--if-not TEXT] [-m WHY] [--words TEXT] experiment positional arguments: experiment options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --if-holds TEXT the action taken when it holds --if-not TEXT what to do when it does not -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb variant ```text usage: rb variant [-h] <add> ... positional arguments: <add> add add a variant to an experiment options: -h, --help show this help message and exit ``` ### rb variant add ```text usage: rb variant add [-h] [--json] [--dry-run] --role {baseline,candidate,control,ablation} [--note NOTE] [--amend] [-m WHY] [--words TEXT] experiment name positional arguments: experiment name options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --role {baseline,candidate,control,ablation} --note NOTE --amend the experiment is frozen: add it anyway (with --why); the amendment is recorded -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb note ```text usage: rb note [-h] <add> ... positional arguments: <add> add add a note, kept verbatim with who wrote it options: -h, --help show this help message and exit ``` ### rb note add ```text usage: rb note add [-h] [--json] [--dry-run] [-e EXPERIMENT] [--words TEXT] text positional arguments: text options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would -e EXPERIMENT, --experiment EXPERIMENT the experiment it is about (default: the whole project) --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb metric ```text usage: rb metric [-h] <add> ... positional arguments: <add> add declare a metric options: -h, --help show this help message and exit ``` ### rb metric add ```text usage: rb metric add [-h] [--json] [--dry-run] [--unit UNIT] [--minimize | --maximize] [--alias ALIAS] [--description DESCRIPTION] name positional arguments: name options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --unit UNIT --minimize lower is better (an error, a loss, a latency) --maximize higher is better (an accuracy, a score) --alias ALIAS another name runs may use for it (repeatable), e.g. cross_entropy --description DESCRIPTION ``` ## rb spec ```text usage: rb spec [-h] <set|verify|vary> ... positional arguments: <set|verify|vary> set give a setting's value and source (checked when it can be), record it as unknown, or read many from a config with --from verify check each setting's source again and record what it states; verified only if it states the value vary declare settings the variants differ in on purpose; any other difference is a confound options: -h, --help show this help message and exit ``` ### rb spec set ```text usage: rb spec set [-h] [--json] [--dry-run] [--unknown] [--per-run] [--design-factor | --sample-factor] [--source SOURCE] [--quote QUOTE] [--term TERM] [--commit COMMIT] [--locator LOCATOR] [--variant VARIANT] [--optional | --required] [--cited VALUE] [--cited-source CITED_SOURCE] [--from FILE] [--keys KEYS] [--note NOTE] [--no-verify] [--amend] [-m WHY] [--words TEXT] experiment [name] [value] positional arguments: experiment name the setting; <variant>.<name> for one variant's own value value a number, true/false, text, or a JSON list; optional with file#key or run: sources options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --unknown nobody has found this value yet --per-run each run gives its own value (a seed): --set seed=2 on rb run add --design-factor a fixed design choice, not an independent sample; a new setting becomes per-run, and an existing one must already be per-run --sample-factor clear this setting's design-factor declaration --source SOURCE where the value comes from: file#key (YAML/JSON/TOML), database#table[column=value].column (SQLite), file:LINE, run:PATH#/pointer, https://..., or note:text --quote QUOTE the exact text in a file or url source that states it --term TERM the word the source uses for it on a prose line, e.g. "learning rate" --commit COMMIT read the file at this git commit instead of the working tree --locator LOCATOR how a reader finds it, e.g. "§4.2" or "Table 3" --variant VARIANT set it for this variant only (same as <variant>.<name>) --optional an unknown value is a note, not an integrity problem of the findings --required it does (the default for a new setting) --cited VALUE the value the cited source (a paper, its evaluation code) used; a difference makes cited criteria not comparable --cited-source CITED_SOURCE where the cited value comes from --from FILE record settings from a YAML/JSON/TOML config, each by key path, with --keys --keys KEYS with --from: the keys that define the experiment, glob patterns, e.g. "optim.*,model.depth" --note NOTE --no-verify record it without checking the source now --amend the experiment is frozen: change it anyway (with --why); the amendment is recorded -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ### rb spec verify ```text usage: rb spec verify [-h] [--json] [--dry-run] experiment [names ...] positional arguments: experiment names settings (default: every setting with a source) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would ``` ### rb spec vary ```text usage: rb spec vary [-h] [--json] [--dry-run] [--amend] [-m WHY] [--words TEXT] experiment names [names ...] positional arguments: experiment names setting names (or comma-separated) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --amend the experiment is frozen: change it anyway (with --why); the amendment is recorded -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb criterion A criterion is the rule a result must meet. ```text usage: rb criterion [-h] <add|file|relate> ... positional arguments: <add|file|relate> add declare a criterion; declare it before the evidence that tests it file declare which question a criterion bears on and what meeting it would mean relate declare, with its reason, that a comparison reverses another and how their arms align; rb reads whether it does options: -h, --help show this help message and exit ``` ### rb criterion add ```text usage: rb criterion add [-h] [--json] [--dry-run] -e EXPERIMENT [-q QUESTION] [--when-met WHEN_MET] --metric METRIC [--subject SUBJECT] [--reference REFERENCE] [--estimand ESTIMAND] [--where NAME=VALUE] [--subject-where NAME=VALUE] [--reference-where NAME=VALUE] [--at-most AT_MOST] [--at-least AT_LEAST] [--equals TARGET] [--tolerance TOLERANCE] [--better] [--noninferior MARGIN] [--equivalent MARGIN] [--each] [--bound] [--count K] [--best] [--of N] [--relative] [--means] [--role {primary,secondary,exploratory,gate}] [--min-n MIN_N] [--noise NOISE] [--source SOURCE] [--quote QUOTE] [--term TERM] [--commit COMMIT] [--locator LOCATOR] [--note NOTE] [--amend] [-m WHY] [--words TEXT] [--id ID] statement positional arguments: statement options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would -e EXPERIMENT, --experiment EXPERIMENT the experiment whose runs test it -q QUESTION, --question QUESTION the question this criterion bears on (repeatable) --when-met WHEN_MET your interpretation for these questions when the rule is met, verbatim --metric METRIC what is measured: val_loss, int8.val_loss, candidate.val_loss, or change.val_loss (the candidate against the baseline); with --subject/--reference, the bare metric --subject SUBJECT the arm it is about: a variant, a role, or an arm an analysis names (default: the only candidate) --reference REFERENCE the arm it is compared against; without one it reads a level --estimand ESTIMAND what is estimated, when an analysis reports it that way: pooled, per_class (default: rb's reading of the runs, a mean over units) --where NAME=VALUE select exact per-run setting values; repeat to require all (where) --subject-where NAME=VALUE select exact per-run setting values; repeat to require all (subject-where) --reference-where NAME=VALUE select exact per-run setting values; repeat to require all (reference-where) --at-most AT_MOST --at-least AT_LEAST --equals TARGET, --within TARGET with --tolerance: holds when |observed - TARGET| <= tolerance --tolerance TOLERANCE --better the subject is better than the reference: the interval clear of 0 on the better side --noninferior MARGIN the subject is no worse than the reference by more than MARGIN, read on the interval --equivalent MARGIN the difference lies inside ±MARGIN, read on the interval --each a threshold every independent unit must meet, not only the estimate --bound a threshold on the interval's bound (the upper for --at-most, the lower for --at-least) --count K a threshold at least K independent units must meet ("19 of 20 runs below 0.890"; with --of N) --best a threshold the best unit must meet, best in the threshold's direction ("best of 5 runs reaches 0.889"; with --of N) --of N with --count or --best: how many units the rule is of; runs that hold another number leave it undetermined --relative read the relative change, (subject - reference) / reference, its threshold a percent ("improves val_bpb by 22.5%": --at-most -22.5) --means with --source: the cited number is a difference of the arms' means over all their units, and rb reads that --role {primary,secondary,exploratory,gate} primary (what the work claims), secondary, exploratory, or gate (a priority policy: a miss parks a line, it never refutes it) --min-n MIN_N independent units it takes (default 1) --noise NOISE run-to-run spread: an estimate this close to the rule is said to be within the noise --source SOURCE where the cited number (a paper's table) comes from: file#key (YAML/JSON/TOML), file:LINE, run:PATH#/pointer, https://..., or note:text --quote QUOTE the exact text that states the number, or the ordering a threshold of 0 cites; a file source must state it or the criterion is refused --term TERM the word the source uses for it on a prose line, e.g. "learning rate" --commit COMMIT read the file at this git commit instead of the working tree --locator LOCATOR how a reader finds it, e.g. "§4.2" or "Table 3" --note NOTE --amend the experiment is frozen: add it anyway (with --why); the amendment is recorded -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason --id ID ``` ### rb criterion file ```text usage: rb criterion file [-h] [--json] [--dry-run] [--when-met WHEN_MET] [--remove] [--amend] -m WHY [--words TEXT] criterion question positional arguments: criterion question options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --when-met WHEN_MET your interpretation when the rule is met, verbatim; omitted keeps the current words --remove retract this question declaration, keeping its history --amend change the frozen declaration with the reason recorded -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ### rb criterion relate ```text usage: rb criterion relate [-h] [--json] [--dry-run] [--orientation {same,opposite}] [--remove] [--amend] -m WHY [--words TEXT] criterion {reverses} target positional arguments: criterion {reverses} target options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --orientation {same,opposite} same: corresponding subject/reference order; opposite: swap the target's order --remove withdraw the relation, keeping its attribution --amend change a frozen relation with the reason recorded -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb run rb stores each run with a receipt: what ran and on which commit. ```text usage: rb run [-h] <add> ... positional arguments: <add> add record a run: numbers from any evaluator, or a file options: -h, --help show this help message and exit ``` ### rb run add ```text usage: rb run add [-h] [--json] [--dry-run] [--variant VARIANT] [--from FILE] [--keys KEYS] [--table TABLE] [--where COLUMN=VALUE] [--every-row] [--select COLUMN] [--config CONFIG] [--set NAME=VALUE] [--input RESOURCE[:ROLE]] [--output RESOURCE[:ROLE]] [--host NAME=VALUE] [--commit COMMIT] [--tree TREE] [--artifact ARTIFACT] [--link NAME=URL] [--command COMMAND_TEXT] [--command-source FILE[:LINE]] [--again] [--replaces RUN] [--repeats RUN] [--produced-at WHEN] [-m WHY] [--words TEXT] [--note NOTE] experiment [NAME=VALUE ...] positional arguments: experiment NAME=VALUE results, e.g. int8.val_loss=5.64 latency_ms=25.1 options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --variant VARIANT the variant these numbers are for: val_loss=5.64 becomes <variant>.val_loss --from FILE a JSON/YAML/TOML object of metric names to numbers (nested keys join with dots), a CSV, TSV or JSON Lines table with --where, or a SQLite database with --table --keys KEYS with --from: only these numbers, glob patterns, e.g. "summary.*" (the file is still hashed whole) --table TABLE with --from a SQLite database: the table that holds the run's row (the database is read, never written) --where COLUMN=VALUE with --table, or a CSV, TSV or JSON Lines table: the row's columns and their values, e.g. precision=int8,seed=1; exactly one row must have them all --every-row with --table, or a CSV, TSV or JSON Lines table: record every row --where picks, each as its own run (a SQLite row kept by its rowid), its @<column> values from its own row --select COLUMN with --table, or a CSV, TSV or JSON Lines table: only these of the row's columns that name a declared metric (default: all of them) --config CONFIG the run's own resolved config (Hydra, W&B config.yaml, hparams.yaml): checks it ran with the spec --set NAME=VALUE a per-run setting this run used, e.g. seed=2; with --table, seed=@seed takes the row's seed column, and seed=@configs.seed (or @configs[config_id].seed) the seed of the row of configs that table links --input RESOURCE[:ROLE] a resource the run read, e.g. imagenet-val:eval (repeatable) --output RESOURCE[:ROLE] a resource the run made, e.g. int8-ckpt:checkpoint (repeatable) --host NAME=VALUE where it ran, as you say it: provider=runpod gpu=A100 (repeatable) --commit COMMIT the commit that produced the numbers --tree TREE the git tree of the exact code that produced them, when it had uncommitted changes (rb journal run keeps it; rb reconcile passes it) --artifact ARTIFACT, --file ARTIFACT an output whose hash goes into the run (repeatable) --link NAME=URL where else it lives, e.g. wandb=https://... (repeatable) --command COMMAND_TEXT the command that produced the numbers (recorded as given; rb does not run it) --command-source FILE[:LINE] the log line that records that command's launch, for a run made before it was recorded; rb checks the line holds it --again record numbers identical to a recorded run's as a second execution (with --why) --replaces RUN this run replaces RUN, an invalid run of the same experiment: RUN is retracted with --why and the two are linked --repeats RUN this run re-checks RUN: it joins that run's unit and adds none --produced-at WHEN when the output existed, when that is earlier than now (ISO date or time), or unknown; with --table, @COLUMN reads it from the row -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason --note NOTE ``` ## rb analysis An analysis is a computation over saved outputs. ```text usage: rb analysis [-h] <add> ... positional arguments: <add> add record an analysis: its comparisons, what it read, its method, and where its numbers are stated options: -h, --help show this help message and exit ``` ### rb analysis add ```text usage: rb analysis add [-h] [--json] [--dry-run] [--compare SUBJECT/REFERENCE:METRIC[@ESTIMAND]=EST[LO,HI]] [--from FILE] [--title TITLE] [--reads RUN|ANALYSIS] [--input RESOURCE[:ROLE]] [--artifact ARTIFACT] [--method METHOD] [--unit UNIT] [--units UNITS] [--draws DRAWS] [--level LEVEL] [--source SOURCE] [--quote QUOTE] [--commit COMMIT] [--locator LOCATOR] [--supersedes ANALYSIS] [--produced-at WHEN] [--command COMMAND_TEXT] [-m WHY] [--words TEXT] [--note NOTE] [--id ID] experiment positional arguments: experiment options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --compare SUBJECT/REFERENCE:METRIC[@ESTIMAND]=EST[LO,HI] one comparison it produced (repeatable), e.g. new/old:score=0.12[0.07,0.17] --from FILE its output file listing the comparisons (JSON/YAML/TOML); hashed, and read as computed --title TITLE --reads RUN|ANALYSIS runs or analyses it read (repeatable or comma- separated) --input RESOURCE[:ROLE] a resource it read, e.g. predictions:eval (repeatable) --artifact ARTIFACT, --file ARTIFACT an output whose hash goes into the analysis (repeatable) --method METHOD how the intervals were made, e.g. "paired bootstrap over scenes" --unit UNIT the independent unit it resampled, e.g. "scene" --units UNITS how many of them --draws DRAWS bootstrap draws, when it resampled --level LEVEL the intervals' level (default 0.95) --source SOURCE the document that states the numbers (a README, a report saved as text): rb finds the line that states each --quote QUOTE the exact row that states them --commit COMMIT read the document at this git commit --locator LOCATOR how a reader finds it, e.g. "Table 2" --supersedes ANALYSIS an earlier analysis this one replaces (with --why) --produced-at WHEN when it was computed, when earlier than now, or unknown --command COMMAND_TEXT the command that computed it (recorded as given; rb does not run it) -m WHY, --why WHY why it supersedes the earlier analysis --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason --note NOTE --id ID ``` ## rb resource ```text usage: rb resource [-h] <add|locate|expose> ... positional arguments: <add|locate|expose> add record a resource: its identity, its locations, and how to get it again locate record another place a resource's bytes are (or were) expose record that someone looked at a dataset: inspected it, selected on it, evaluated on it, trained on it options: -h, --help show this help message and exit ``` ### rb resource add ```text usage: rb resource add [-h] [--json] [--dry-run] --type {dataset,checkpoint,code,environment,artifact,other} --name NAME [--version VERSION] [--subset SUBSET] [--file FILE] [--sha256 SHA256] [--digest DIGEST] [--at PROVIDER:REF] [--source SOURCE] [--url URL] [--recipe RECIPE] [--note NOTE] id positional arguments: id a short id you choose, e.g. imagenet-val options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --type {dataset,checkpoint,code,environment,artifact,other} --name NAME what it is, e.g. "ImageNet-1k validation" --version VERSION --subset SUBSET which part, e.g. "training split, images 0..199" --file FILE a local file: hashed now, and recorded as its first location --sha256 SHA256 the sha256 of the file, when it is one file --digest DIGEST any other fingerprint: a provider's digest, a dataset hash --at PROVIDER:REF where its bytes are (repeatable); providers: local, url, git, wandb, mlflow, s3, gcs, hf, runpod, other --source SOURCE where it comes from, in words --url URL where to get it again --recipe RECIPE the command or steps that rebuild it --note NOTE ``` ### rb resource locate ```text usage: rb resource locate [-h] [--json] [--dry-run] [--note NOTE] id {local,url,git,wandb,mlflow,s3,gcs,hf,runpod,other} ref positional arguments: id {local,url,git,wandb,mlflow,s3,gcs,hf,runpod,other} ref a path, a uri, an artifact name, a pod id options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --note NOTE ``` ### rb resource expose ```text usage: rb resource expose [-h] [--json] [--dry-run] [--at AT] [-m WHY] [--words TEXT] id {inspected,selected,evaluated,trained} positional arguments: id {inspected,selected,evaluated,trained} options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --at AT when it happened, when earlier than now (ISO date or time), or unknown; kept as reported -m WHY, --why WHY what was looked at, and why --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` --- # Commands: Calls https://rabbitbrain.ai/docs/reference/cli/calls/ A call is a decision on the work, such as accepting a result. Generated from the `--help` of rabbit-brain 0.8.4. ## rb freeze Locks an experiment's spec and criteria (the rules its results must meet) before the evidence, and records who locked it. ```text usage: rb freeze [-h] [--json] [--dry-run] [--amend] [-m WHY] [--words TEXT] experiment positional arguments: experiment options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --amend the experiment is frozen: adopt it as it is now (with --why) -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` ## rb decide Records a call on a criterion, an experiment, a question or a setting, with who made it and why. In an agent session it is only proposed until you confirm it, which makes it yours. ```text usage: rb decide [-h] [--json] [--dry-run] -m WHY [--words TEXT] [--revisit REVISIT] [--past] [--when TEXT] [--source FILE[:LINE]] [--quote TEXT] subject {accept,reject,investigate,continue,park,stop,pivot,ship} positional arguments: subject {accept,reject,investigate,continue,park,stop,pivot,ship} options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason --revisit REVISIT when to look again, e.g. "when the new images are in" --past a call made before it was recorded: it reads Retrospective, dated as --when says --when TEXT with --past: when the call was made, as said (a date, or words); unknown without it --source FILE[:LINE] with --past and --quote: the document that states the call; rb says what git shows of it, and a past call from an agent session still waits for rb confirm --quote TEXT with --source: the document's own words that state the call, naming what it is on; rb checks the file holds them ``` ## rb confirm Confirms what an agent session left waiting: a proposed call, a past call or an experiment's design. Run it in your own terminal. With no id, it lists what waits. ```text usage: rb confirm [-h] [--json] [--dry-run] [--all] [ID ...] positional arguments: ID the calls to confirm, as rb status lists them, or <experiment>@<hash> for an experiment's design with its gate, as rb confirm lists it; an experiment's id alone lists its design options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --all confirm every call, gate and design that waits as it reads now, after saying each (the command rb confirm ends with confirms them as listed) ``` ## rb retract Takes anything back. It stays on the record but stops counting, and rb says what depended on it. ```text usage: rb retract [-h] [--json] [--dry-run] [--scope MEASUREMENT] -m WHY [--words TEXT] subject positional arguments: subject an id (a note's too), or <experiment>/<setting> options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run run every check the write makes and print what it would write, writing nothing; it exits as the write would --scope MEASUREMENT runs only: supersede these measurements (e.g. --scope latency_ms) and keep the rest counting -m WHY, --why WHY the reason, recorded with the call --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason ``` --- # Commands: Reading it https://rabbitbrain.ai/docs/reference/cli/reading/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb open Shows the record in your browser, live and read-only, on 127.0.0.1 only. ```text usage: rb open [-h] [--json] [--port PORT] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --port PORT port on 127.0.0.1 (default 8787) ``` ## rb ui Opens this project in your terminal. `rb` with no command does the same. ```text usage: rb ui [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb status Shows each criterion (a rule a result must meet) with its finding (what rb computes from the runs), and what is open, with who usually acts on it. ```text usage: rb status [-h] [--json] [--fail-on GATE] [experiment] positional arguments: experiment only this experiment options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --fail-on GATE exit 1 when any of: unmet, unresolved, untested, integrity, undecided, stale, unknown, late, checks, unestablished (repeatable or comma-separated); an edit made outside rb that a finding depends on fails any of them ``` ## rb show Shows one record: a criterion with its finding, an experiment with its spec, a run with its receipt (what ran and on which commit), or a setting with its history. It prints a `.yaml` record file as canonical JSON. ```text usage: rb show [-h] [--json] id positional arguments: id an id, <experiment>/<setting>, or a path to an Rabbit Brain YAML file options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb compare Shows one experiment's table: variants by metrics, with changes against the baseline. ```text usage: rb compare [-h] [--json] [experiment] positional arguments: experiment options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb log Lists every write to the record, newest last: who wrote what, and how. ```text usage: rb log [-h] [--json] [-n N] [id] positional arguments: id only this record, or <experiment>/<setting> options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text -n N how many (default 20) ``` ## rb context Prints the record as one Markdown briefing for a new agent session to read first. ```text usage: rb context [-h] [--json] [--model ID] [--provider NAME] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --model ID an agent's own model, as its provider names it: kept with every later write in this agent session (RB_MODEL gives it for one command) --provider NAME who provides that model, kept the same way (RB_PROVIDER for one command); rb fills in Claude Code's and Codex's when none is given ``` ## rb page Writes the project as one self-contained, read-only HTML file you can send anywhere. ```text usage: rb page [-h] [--json] [--out OUT] [--replace] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --out OUT where to write it (default rb-page.html beside .rb/) --replace write over a file at --out that rb did not write (default: refused, the file left as it is) ``` --- # Commands: Carrying it https://rabbitbrain.ai/docs/reference/cli/carrying/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb export Writes the project's page with a hashed manifest, readable without rb. With `--with-record` it writes the record too, as a portable bundle. ```text usage: rb export [-h] [--json] [--out OUT] [--zip] [--with-record] [--url URL] [--no-repo] [--replace] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --out OUT where to write it (default rb-export/ beside .rb/) --zip write one zip file instead of a directory (default rb- export.zip) --with-record carry the record too, for rb clone: its files and log keep who made each write, agent sessions and the user's words --url URL with --with-record, the public address this will be served from: the page then opens with the exact rb clone command, and a directory export also contains bundle.zip --no-repo leave the code repository's address out of the manifest and page --replace with --zip, write over a file at --out that rb did not write (default: refused, the file left as it is) ``` ## rb clone Starts your own line of work from an exported bundle. Its structure comes in as inherited, and nothing counts until it is reproduced here. ```text usage: rb clone [-h] [--json] [--code ADDRESS] source [dest] positional arguments: source a bundle rb export --with-record wrote: a directory, a .zip, or its address dest where to create it (default ./<project id>) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --code ADDRESS where to get the code the runs need, in place of the repository the bundle names (a copy you can reach, or one the bundle does not name): the record lands in a checkout of it at the bundle's commit ``` ## rb env Checks this machine's environment against the record's runs and any `env.*` settings. ```text usage: rb env [-h] [--json] <command> ... positional arguments: <command> check compare; differences are said, never fatal (exit 0) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ### rb env check ```text usage: rb env check [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` --- # Commands: For agents https://rabbitbrain.ai/docs/reference/cli/for-agents/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb mcp Serves the record to an agent over MCP (stdio), and records every tool call as the agent's. ```text usage: rb mcp [-h] [--agent AGENT] options: -h, --help show this help message and exit --agent AGENT the agent's name in the record (default: the MCP client's name) ``` ## rb skill Prints the agent skill. `rb skill add` installs it for your coding agent. ```text usage: rb skill [-h] [--json] [--claude] [--project] [--cursor] [--agents-md] [--yes] [{add}] positional arguments: {add} add: write it where the agent looks options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --claude ~/.claude/skills/rabbit-brain/ (the default target) --project with --claude: this project's .claude/skills/ instead of the home directory --cursor .cursor/rules/rabbit-brain.mdc in this project --agents-md, --codex a marked section in this project's AGENTS.md, for Codex and anything else that reads it --yes with --agents-md: add rb's section to an AGENTS.md that is here without asking (default: asked once in your terminal; elsewhere left as it is, with what to add) ``` ## rb journal Shows what rb observed in the repository: files that appeared, git moves and wrapped commands. ```text usage: rb journal [-h] [--json] [-n N] <scan|run> ... positional arguments: <scan|run> scan look at the watched places once and journal what changed run run a command and journal what it did: rb journal run -- python eval.py ... options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text -n N how many entries (default 20) ``` ### rb journal scan ```text usage: rb journal scan [-h] [--json] [--hook <agent>] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --hook <agent> run from that agent's session hook (rb init installs it): say nothing, exit 0, and keep the look as that agent's ``` ### rb journal run ```text usage: rb journal run [-h] [--json] ... positional arguments: cmd the command, after -- options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb reconcile Compiles the repository against the record. Each results folder with a report is shown as an experiment, with what its files say (verbatim), its checks and how to reproduce it. ```text usage: rb reconcile [-h] [--json] [--apply] [--no-agents] [--yes] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --apply keep it in the record: add and re-index the experiments, record the runs found, attributed to you --no-agents with --apply where there is no record yet: leave the agents' instruction files alone (default: install rb's, as rb init does) --yes with --apply where there is no record yet: add rb's part to an AGENTS.md or .mcp.json that is here without asking (default: asked once in your terminal; elsewhere left as they are) ``` ## rb compile Lists which existing reports and results files the record can place, and which it cannot. It writes nothing. ```text usage: rb compile [-h] [--json] [paths ...] positional arguments: paths files or directories to compile from options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` --- # Commands: Upkeep https://rabbitbrain.ai/docs/reference/cli/upkeep/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb doctor Checks the record: who you are recorded as, edits made outside rb, merge leftovers and sources outside the repository. ```text usage: rb doctor [-h] [--json] [--restore] [--adopt] [-m WHY] [--words TEXT] [--seen PATH@SHA256] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --restore put back what rb last wrote for every record changed outside rb --adopt take every change made outside rb as it stands (with --why); edits to who wrote or decided what are refused -m WHY, --why WHY the reason, recorded with --adopt --words TEXT the user's words from the chat that this acts on, verbatim (an agent quoting them); kept beside the reason --seen PATH@SHA256 with --adopt: the reviewed files, each pinned to the sha it had (or 'deleted'); any drift since is refused ``` ## rb fmt Rewrites Rabbit Brain YAML files in canonical form, keeping their comments. ```text usage: rb fmt [-h] [--json] [--check] [paths ...] positional arguments: paths files or directories (default: the record) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --check change nothing; exit 1 if any file is not canonical ``` ## rb telemetry Optional usage analytics. It is off until you enable it and never sends research content. ```text usage: rb telemetry [-h] <status|enable|disable|preview|flush> ... positional arguments: <status|enable|disable|preview|flush> status show the local analytics setting and destination without sending anything enable opt in to limited usage analytics on this installation disable stop analytics and delete the local installation ID and pending events preview show the queued payload without sending it flush try sending the pending usage events now options: -h, --help show this help message and exit ``` ### rb telemetry status ```text usage: rb telemetry status [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ### rb telemetry enable ```text usage: rb telemetry enable [-h] [--json] [--project-key PROJECT_KEY] [--host HOST] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --project-key PROJECT_KEY PostHog public project capture key; never a personal API key --host HOST PostHog HTTPS ingestion host (must match the project's region) ``` ### rb telemetry disable ```text usage: rb telemetry disable [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ### rb telemetry preview ```text usage: rb telemetry preview [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ### rb telemetry flush ```text usage: rb telemetry flush [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb config rb's own settings on this machine. ```text usage: rb config [-h] [--json] <auto-update> ... positional arguments: <auto-update> auto-update whether rb checks for a newer rb once a day, and updates itself when it was installed as a tool (uv tool install, pipx); says the setting with no value options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ### rb config auto-update ```text usage: rb config auto-update [-h] [--json] [{on,off}] Whether rb checks PyPI for a newer rb at most once a day: installed as a tool (uv tool install, pipx) it updates itself in the background, and installed with pip it says one line and changes nothing. Never in CI, in an agent session or under DO_NOT_TRACK. With no value it says the setting. positional arguments: {on,off} on or off options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` --- # Commands: Leaving https://rabbitbrain.ai/docs/reference/cli/leaving/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb remove Takes rb out of this checkout, after showing the plan. It removes only what it can tell is rb's: the record, its instructions for agents, its pages and exports, and its refs and folders in git. ```text usage: rb remove [-h] [--json] [--dry-run] [--yes] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --dry-run show the plan (what rb would remove, what it would keep and why), removing nothing --yes remove without asking (default: asked once in your terminal; refused where there is none) ``` --- # Commands: Reference https://rabbitbrain.ai/docs/reference/cli/reference/ Generated from the `--help` of rabbit-brain 0.8.4. ## rb tour A one-minute walkthrough. It builds a small real project in `./rb-tour` and runs the whole loop. ```text usage: rb tour [-h] [--json] [dir] positional arguments: dir where the tour lands (default ./rb-tour; must be new or empty) options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb docs Prints AGENTS.md. With `--errors` it prints the error table. ```text usage: rb docs [-h] [--json] [--errors] [--start] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --errors --start print the start protocol: how an agent records work already done ``` ## rb schema Prints a JSON Schema: `rb schema all` for Rabbit Brain formats 1 and 2 (rb-v2.schema.json), or `rb schema <kind>` for one record's shape. ```text usage: rb schema [-h] [--json] {all,project,question,experiment,metric,criterion,run,analysis,resource,decision,finding,envelope,ir} positional arguments: {all,project,question,experiment,metric,criterion,run,analysis,resource,decision,finding,envelope,ir} options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb version Prints the version. ```text usage: rb version [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text ``` ## rb feedback ```text usage: rb feedback [-h] [--json] [--about WORD [WORD ...]] Prints a link to send feedback on rb: the page shows what it would send before you send it; rb sends nothing. In your own terminal it opens the page in your browser; anywhere else it only prints the link. options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --about WORD [WORD ...] the rb command it is about, by its words (--about run add); its arguments are never sent ``` --- # Commands: rabbitbrain.ai https://rabbitbrain.ai/docs/reference/cli/rabbitbrain-ai/ rabbitbrain.ai is optional. Anyone can make an account at rabbitbrain.ai/signin. Generated from the `--help` of rabbit-brain 0.8.4. ## rb login Signs this machine in with a short code, which you approve on any device where you are signed in. ```text usage: rb login [-h] [--json] [--hub HUB] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --hub HUB the hub's address (default RB_HUB_URL, else rabbitbrain.ai) ``` ## rb logout Revokes this machine's hub credential and removes it from the credentials file. ```text usage: rb logout [-h] [--json] [--hub HUB] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --hub HUB the hub's address (default RB_HUB_URL, else rabbitbrain.ai) ``` ## rb sync Puts a private copy of this record's page in your account, for you to view from anywhere. ```text usage: rb sync [-h] [--json] [--hub HUB] [--with-record] [--watch] options: -h, --help show this help message and exit --json print one JSON object (the envelope) instead of text --hub HUB the hub's address (default RB_HUB_URL, else rabbitbrain.ai) --with-record send the record too: its files and log keep who made each write, agent sessions and the user's words --watch keep the hosted copy current: look every 30 s and sync again when the record changed, at most once a minute, until Ctrl-C ``` --- # The record's format https://rabbitbrain.ai/docs/reference/schema/ Every file in `.rb/` starts with its format, `rb: 1` or `rb: 2`, then its `kind`. One JSON Schema (draft 2020-12) covers both formats. It is served at its `$id`, [rabbitbrain.ai/schema/rb-v2.schema.json](https://rabbitbrain.ai/schema/rb-v2.schema.json), and kept in the repository at [schema/rb-v2.schema.json](https://github.com/rabbit-brain/rb/blob/main/schema/rb-v2.schema.json). ```sh # the whole format, for any JSON Schema validator rb schema all > rb-v2.schema.json # one kind: project, question, experiment, metric, criterion, run, analysis, resource or decision rb schema run # a finding's shape: computed on every read, never stored rb schema finding # what every command prints with --json rb schema envelope # read one record file and check it against the format rb show .rb/experiments/int8-ptq/runs/<id>.yaml ``` ## Rabbit Brain YAML The files use a strict subset of YAML: mappings, lists, text, numbers, `true`, `false`, `null` and comments. rb refuses unquoted values that YAML 1.1 and 1.2 read differently, such as `yes`, `no`, `on`, `off`, dates, numbers with leading zeros, and `1e-4` without a dot. `rb fmt` rewrites files in canonical form and keeps comments. ## Format 1 and format 2 Format 1 does not change. A newer rb that adds a field writes that file in format 2, and an older rb says to upgrade (`E_STATE_NEWER`) instead of reading part of it. A criterion is the rule a result must meet. Format 2 adds four kinds of rule to criteria, and nothing else: - a count: `over: count`, with `count` and `of` - a best of N: `over: best`, with `of` - a relative change: `measure: relative` - a cited difference of means: `measure: means` [Criteria](https://rabbitbrain.ai/docs/concepts/criteria/#four-more-kinds-of-rule) explains each one. A file with `rb: 1` and one of the four, or `rb: 2` and none, is refused (`E_STATE_CORRUPT`). `rb export --with-record` writes `record_format: 2` in the bundle's manifest when a criterion's file is format 2, and 1 otherwise. What each version does: - rb 0.8.0 to 0.8.4 read both formats. rb 0.8.4 writes format 2 only for a criterion that uses one of the four, retracted or not. - rb 0.7.0 and 0.7.1 read format 1 only. Where a record holds a format-2 file, they read the other files but say to upgrade, fail every `--fail-on` gate and write nothing to the record. They refuse to clone a bundle that holds one. Upgrade everyone who shares a record together. ## The vocabulary ```rb-session frame="terminal" $ rb schema ir rb's vocabulary: every record kind, what it means, the command that writes it, and the sources rb requires. question what the work is trying to find out a tree: a sub-question names its parent, and an experiment may be filed under several; where it stands is read from what is below it, never a child's negative as its own write: rb question add "<what are you trying to find out?>" --parent <q> provenance: wording as asked experiment what is tested its purpose or hypothesis, one baseline and variants with roles, varies naming the intended differences; planned until its first run; an undeclared difference between variants is a confound write: rb experiment add "<title>" --hypothesis "<h>" --baseline <b> --candidate <c> --varies <s> provenance: declared, then checked against its settings variant one arm of an experiment baseline, candidate, control or ablation; its own settings override the shared ones write: rb variant add <exp> <name> --role <r> provenance: role declared setting what the experiment changes or depends on a value with where it comes from; population.* declares the counting structure and never blocks write: rb spec set <exp> <name> <value> --source file#key provenance: REQUIRED: a source rb can read, or --unknown, or a vouch (rb decide <exp>/<name> accept) criterion what would count on one comparison a rule over one comparison (subject, reference, metric, estimand), with a role (primary, secondary, exploratory, gate), declared by someone at a time before the evidence it reads; explicit question links may keep the author's verbatim interpretation when met, never an answer inferred by rb write: rb criterion add "<c>" -e <exp> -q <question> --metric <m> --subject <a> --reference <b> --better provenance: declared by whoever wrote it, with the user's words when an agent gives them; question links snapshot the question and retain author, time, reason, session and revisions; REQUIRED for cited numbers: --source --quote --locator, checked against the file run one execution, and what it produced results with a receipt: actor, chat session, the agent (its name, version, provider and model), time, repository state, command, environment, the resources read and made, when its output existed; the same output recorded twice is one run; a repeat checks a run and adds no unit; a wrong run is retracted (whole, or --scope for named measurements), never deleted write: rb run add <exp> --from <file> --set seed=N --input <resource> --config <resolved> provenance: REQUIRED: the receipt is taken when it is recorded; --from and --config are read and hashed; typed numbers are marked typed analysis a computation over saved outputs the comparisons it produced, each with its estimate and interval, the unit it resampled, and where it came from; a reanalysis is an analysis, never a run; a later one supersedes it with a reason, and its readings stay as history write: rb analysis add <exp> --compare "a/b:m=est[lo,hi]" --method "<m>" --unit "<u>" --units N --source <file> provenance: REQUIRED: computed from an output file rb hashed, or reported from a document rb checked, or marked typed comparison subject against reference on a metric the unit of state: an estimate, its interval when there is one, what it resampled; changing the metric, the reference or the estimand is a different comparison write: read by a criterion; produced by an analysis or by rb from the runs provenance: an analysis, or rb's reading of the runs result a number a run reports named <variant>.<metric>, change.<metric> (candidate minus baseline) or plain; compared across variants in rb compare write: NAME=VALUE on rb run add, or --from <file> provenance: the run's receipt resource a dataset, checkpoint, code, environment or artifact identity (name, version, hash), locations (any provider), and how to get it again; availability is computed: present, changed, remote, reconstructible or lost; a run records what it found when it read or made one write: rb resource add <id> --type dataset --name "<n>" --url <u> provenance: declared; a local file is hashed when recorded metric what is measured, by name unit, direction, aliases; a result naming an alias is read through it write: rb metric add <m> --unit <u> --minimize provenance: declared finding a criterion read on axes, computed result, rule, timing, units, integrity, review and history, computed from the runs and analyses on every read, never stored; no single word stands for it write: read it: rb show <criterion> provenance: derived decision a call, with what it was made on accept, reject or investigate a finding, an experiment, a question or a setting; continue, park, stop, pivot or ship a line of work; read against the readings it was made on, with when to revisit; one recorded in an agent session is proposed until the user confirms it (rb confirm), and a past one (--past) is Retrospective write: rb decide <subject> <action> -m "..." --revisit "..." provenance: REQUIRED: who made it and why, with the chat session when known and the user's words verbatim when an agent acts on them; for a past call, when it was made and the document that states it note words about an experiment or the project an assumption, a caveat, context; kept verbatim with who wrote it; changes no finding write: rb note add "<text>" -e <exp> provenance: text as written relation how one experiment stands to another reproduces, extends, follows_from or compares_with an experiment here, a uid, or a published manifest; structure, not evidence write: rb experiment relate <exp> reproduces <target> provenance: declared ``` --- # llms.txt https://rabbitbrain.ai/docs/agents/llms-txt/ If you are an agent in a repository with a `.rb/` folder, start with `rb context --model <your model id>`, not these pages. ## These docs, as text | Address | What it is | | --- | --- | | [/docs/llms.txt](https://rabbitbrain.ai/docs/llms.txt) | An index of every page, with a one-line description each. | | [/docs/llms-full.txt](https://rabbitbrain.ai/docs/llms-full.txt) | Every page but the Changelog, in one plain-text Markdown file. | [rabbitbrain.ai/llms.txt](https://rabbitbrain.ai/llms.txt) is rb's own summary for agents. ## Built into rb The manual ships with rb and matches the installed version: ```sh rb docs # the whole manual: every rule, every command, the JSON envelope, every error and its fix rb docs --start # the protocol for starting the record in a project already under way rb docs --errors # the error table rb skill # the skill rb installs for coding agents ``` Over MCP (`rb mcp`), the manual is the resource `rb://docs`, beside `rb://start`, `rb://context`, `rb://status`, `rb://schema` and `rb://format`. ## What an agent must not do - Compute a finding (the result rb works out from the runs), an error, a ranking or a regression itself. Record the numbers, and read the finding with `rb show`. - Edit `.rb/` by hand. - Record a call (a decision on the work) the user did not ask for. - Run `rb confirm`, which approves a call or a design. Only the user runs it, in their own terminal. - Set or unset `RB_ACTOR` or `RB_HUB_TOKEN`, or read the credentials file `rb login` writes. See [Auth](https://rabbitbrain.ai/docs/agents/auth/). --- # Auth https://rabbitbrain.ai/docs/agents/auth/ Only `rb login`, `rb sync` and `rb logout` use an account. They talk to rabbitbrain.ai unless you name another hub with `--hub <address>` or `RB_HUB_URL`. Anyone can make an account at [rabbitbrain.ai/signin](https://rabbitbrain.ai/signin). ## Log in Before you log in, `rb sync` sends nothing: ```rb-session frame="terminal" $ rb sync rb sync: No credential for rabbitbrain.ai on this machine. → Run rb login once per machine, or set RB_HUB_TOKEN for CI. ``` `rb login` signs this machine in: 1. The terminal shows a code: ```rb-session frame="terminal" $ rb login First, this terminal gets a code: NN6T-D43Q Open https://rabbitbrain.ai/link?code=NN6T-D43Q on any device; it signs you in first if needed, and approve the code. ``` 2. The person approves it at the hub's `/link` page, on any device. The credential is theirs, whoever started `rb login`. 3. rb stores the token in `~/.config/rabbit-brain/credentials.json` with mode 0600, or under `$RB_CONFIG_DIR`. For CI, `RB_HUB_TOKEN` overrides the file. `rb logout` revokes the credential on the hub and removes it from this machine. With nothing stored: ```rb-session frame="terminal" $ rb logout Nothing stored for <https://rabbitbrain.ai>; nothing to do. ``` ## Sync `rb sync` uploads the record's page as a private copy in your account. `rb sync --with-record` also sends the record, including who made each write and the words the user gave. `rb sync --watch` syncs again after changes, at most once a minute, until Ctrl-C. A hub's refusal comes back as `E_HUB_REFUSED`, with its reason, and a hub rb cannot reach as `E_HUB_UNREACHABLE`. Either way, nothing on this machine changes. ## For an agent An agent may run `rb login` for the user, who approves the code in their own browser. An agent must not: - read the credentials file - print or copy a token, or put one in chat, logs, commits or files - set or unset `RB_HUB_TOKEN` or `RB_ACTOR` - run `rb sync --with-record` unless the user asks to send the record ---