Catch the regressions your metrics miss.
Rabbit Brain reviews a candidate checkpoint against the current one, case by case, and reads each model’s own refinement iterations to flag the answers it never settled on. Every run leaves a receipt a colleague can verify, whether you ran it or your coding agent did.
Apache-2.0. Runs on your machine; nothing leaves it. Or try the workspace with example data. Tell your agent: “Review candidate checkpoint B against A on this case set with Rabbit Brain.”
turn-026Tight corner
Error fell by 1.10 px. But the candidate made 32% of its refinement in the last third of its iterations and its update grew again 3 times; the current model settled at 4%. The better number is not a settled answer.
Two checkpoints in. A ranked queue and a receipt out.
Point it at the current and candidate checkpoints and a case set, or import the per-case results your evaluator already produces. One label-free number per refinement iteration comes from a forward hook; no change to your model code.
Run or import
rb run --baseline A --candidate B evaluates both checkpoints on your cases and records the trajectories for you (RAFT-family optical flow today, a supported or custom adapter). Or rb import results.json with per-case errors from your own evaluator.
See what needs a decision
Worse on error and still moving at the end comes first, then worse on error, then the cases that improved on error and never settled. That third group is the one a mean would have let through. The stability half reads the model’s own refinement, so it also runs on cases you have no ground truth for.
Keep the check, keep the receipt
rb check save keeps an error limit and a settled-answer requirement per case; rb check run fails CI on the next candidate. Every run writes report.md and record.json with the exact command, checkpoint hashes, seeds and environment.
Not sure it fits your model? Describe your setup and you get back a configuration, an integration ladder, and either a built-in adapter or the honest shape of one with the parts only you can write marked out. It asks for no account, no checkpoints and no data, and nothing is sent anywhere. Supported today: a RAFT-family checkpoint pair on KITTI-style data through the built-in adapter, another model through a custom adapter you write (one file, six methods), or any paired, lower-is-better error metric with a unit (endpoint error in px, absolute depth error in cm) imported from your evaluator. Built for coding agents: rb docs prints the manual, every command speaks JSON with --json, and every error carries a code and a fix.
The model tells you when it isn’t sure. Its final answer doesn’t.
A converged model keeps shrinking its updates; one that is still moving at the end is unreliable on that case whatever the final number says. On the official RAFT checkpoints, the size of the last refinement steps predicts a case’s error at a Spearman rank correlation of 0.88 to 0.92, on 200 cases for each of the four public checkpoints. The signal itself needs no labels; measuring that it works did. Rabbit Brain turns that into a paired test: the candidate is a trajectory regression on a case when it was still moving more than the current model was.
Three real release reviews, receipts in the repository, all three re-run on different hardware and a different torch build. Every verdict came out the same, and six of 200 cases sat close enough to a limit that they changed sides. A report marks those six borderline from the first machine’s numbers alone, before anyone re-runs. It is also the reason the built-in limits are a starting point and a scorer fitted to your model is the paid step: a generic threshold had cases inside its own noise. One benchmark, one setting; your model, data and limits decide your numbers. The research reference is added when the paper is public.
What the receipt says
Verdict: Not ready: 1 error regression. Start with 000145_10 (5.48 → 5.89 px). The other 199 cases improved or stayed within 0.3 px. Checkpoints identified by sha256, RAFT at commit 2888e15, torch 2.4.1+cu124, seeds and the exact command recorded.
Two more reviews from the same night: raft-things → raft-kitti, 5.40 → 0.61 px, one case improved on error but did not settle; raft-things → raft-small, 155 regressions, the 39 worst ranked first by the trajectory test. All three are in examples/raft-kitti in the repository.
And a check on the receipt itself: all three reviews re-run on different hardware and a different torch build. On its reference cases the adapter matched RAFT’s own evaluation code to within 0.001 relative on all four checkpoints, which is the agreement the run enforces before it reports anything, and every verdict and error regression came out the same case for case. Six of the 200 cases sat close enough to a limit to change sides, which is why a report marks the findings that turn on a margin that thin.
The tool is free and stays useful on its own.
Your runs, findings, receipts and checks are plain files in your repository. They are yours, we never hold them, and nothing below unlocks them. What you can pay for is work we keep doing around that history.
Your checkpoints, reviewed.
pip install rabbit-brain. Run both checkpoints or import your evaluator’s results. Ranked queue, plain-language findings, saved checks, receipts.
- Error regressions and trajectory regressions, ranked, with the numbers beside them
- Saved checks that follow case IDs across releases and fail CI
- Made for coding agents:
rb docs,--jsoneverywhere, exit codes, error codes with fixes
Generic limits: how much the error may rise, how much more the candidate may still be moving at the end than the current model, how much of its refinement may land in the last third, how many times its update may grow again. Shown against your data’s own distribution in every report, so you can see where a limit sits before you trust it. This is the whole tool; there is no held-back version, and the files it writes stay in your repository whether you ever pay us anything or not.
We run the comparison. You get the code.
We fit a trajectory scorer to your model, compare it with your current baseline at equal retained coverage, measure the overhead, and hand everything over.
- Phase A · US$1,500. Compatibility & protocol. Confirm the hook works on your model; agree the protocol, or stop with a documented reason.
- Phase B · US$3,500. Evaluation & handover. Fitted scorer, measured comparison, adapter, protocol and report.
- Rerun it on the next checkpoint without us.
One offline workflow, one frozen model. You pay for the investigation and handover; your data determines the outcome, and a negative finding is a valid one. Most teams should run the free tool for a few checkpoints first: the limits it ships with are generic, and the case for fitting them to your model is something your own receipts will show you.