Criteria
A criterion is the rule a result must meet. Each one covers a single comparison, such as INT8 against FP32. rb records who declared it and when, and checks it against the runs each time the record is read. The outcome is the criterion’s finding.
rb criterion add "INT8 loses at most 0.5 points of top-1 against FP32" \ -e int8-ptq -q q-int8 --metric change.top1 --at-least -0.5 --min-n 3 --why "docs/plan.md:3"What it compares
Section titled “What it compares”--metric names a number the runs report:
--metric | Reads |
|---|---|
top1 | the number named top1, or the only candidate’s when runs report it per variant |
int8.top1 | the int8 variant’s number |
candidate.top1 | the candidate’s number |
change.top1 | the candidate’s number minus the baseline’s, from the same run |
top1 with --subject fp32 --reference int8 | FP32’s number against INT8’s |
Record each variant’s own numbers, such as {"fp32": {"top1": 91.71}, "int8": {"top1": 91.6}}. rb works out change.top1 itself.
The rule
Section titled “The rule”The estimate is the mean over units. A unit is an independent repeat, such as a seed.
| Rule | Holds when |
|---|---|
--at-most X, --at-least X | the estimate is at most X, or at least X |
--equals X --tolerance T | the estimate is within T of X |
--each | every unit meets the threshold |
--bound | the interval’s bound meets the threshold |
--better | the interval is clear of 0 on the better side |
--noninferior M | the interval’s worse bound is inside M |
--equivalent M | the interval is inside ±M |
--better, --noninferior and --equivalent need an interval, which rb gives from 3 units. --better and --noninferior also need to know which way is better, or rb refuses them: rb metric add top1 --maximize.
--min-n 3 sets the fewest units the criterion needs. With fewer, it is undetermined.
Four more kinds of rule
Section titled “Four more kinds of rule”rb 0.8.4 has four more kinds of rule:
| Rule | Reads |
|---|---|
--count K --of N | whether at least K of the N units meet the threshold. Undetermined unless the runs hold N units. |
--best --of N | the best unit’s own value, with --at-most or --at-least. It has no interval. |
--relative | the relative change of the arms’ means, (subject - reference) / reference, in percent. The threshold is a percent too. |
--means, with --source and --quote | the cited number as a difference of the arms’ means over all their units |
Only a criterion with one of these rules is saved in format 2 (rb: 2). See The record’s format. Read a rule over seeds has worked examples.
Where the threshold comes from
Section titled “Where the threshold comes from”A threshold comes from a file or from you, never from the agent:
- A plan:
--why "docs/plan.md:3". rb refuses the criterion if the line is missing, is a comment, or does not state the threshold. - A paper or other document:
--source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6". rb checks that the file holds the quote and that it states the number. - Your words:
--words "...", kept verbatim. Words an agent passes show aswords (as given), since rb cannot know you said them.
--role primary, the default, is what the work claims. secondary and exploratory are context. A gate only steers where effort goes. A missed gate refutes nothing. It asks you to decide whether to park or continue the line of work.
When it was declared
Section titled “When it was declared”Every finding says when its criterion was written, compared with its evidence:
| rb says | When |
|---|---|
| declared before the evidence | it was written before its runs were produced |
| declared after the evidence existed | some or all of its runs already existed. rb says how many. |
| declared after the data was seen | the data it evaluates on was looked at first |
| order unknown | some evidence has no known time, such as numbers typed in with no --produced-at |
Evidence that existed first still counts, but rb never reads it as if the criterion came first. Declare the criterion before the runs shows both timings side by side.
Freezing
Section titled “Freezing”rb freeze <experiment> -m "before the runs" locks the experiment’s variants, settings and criteria, and records their hash. After that, a change fails with E_FROZEN unless the command adds --amend --why "...". rb keeps each amendment with who made it.
Which question it answers
Section titled “Which question it answers”-q <question> links a criterion to a question, and only linked criteria count for that question. --when-met "..." keeps your words on what meeting the rule would mean. If the rule misses, rb does not claim the opposite.
Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0