Skip to content

Criteria

A criterion is the rule a result must meet. Each one covers a single comparison, such as INT8 against FP32. rb records who declared it and when, and checks it against the runs each time the record is read. The outcome is the criterion’s finding.

Terminal window
rb criterion add "INT8 loses at most 0.5 points of top-1 against FP32" \
-e int8-ptq -q q-int8 --metric change.top1 --at-least -0.5 --min-n 3 --why "docs/plan.md:3"

--metric names a number the runs report:

--metricReads
top1the number named top1, or the only candidate’s when runs report it per variant
int8.top1the int8 variant’s number
candidate.top1the candidate’s number
change.top1the candidate’s number minus the baseline’s, from the same run
top1 with --subject fp32 --reference int8FP32’s number against INT8’s

Record each variant’s own numbers, such as {"fp32": {"top1": 91.71}, "int8": {"top1": 91.6}}. rb works out change.top1 itself.

The estimate is the mean over units. A unit is an independent repeat, such as a seed.

RuleHolds when
--at-most X, --at-least Xthe estimate is at most X, or at least X
--equals X --tolerance Tthe estimate is within T of X
--eachevery unit meets the threshold
--boundthe interval’s bound meets the threshold
--betterthe interval is clear of 0 on the better side
--noninferior Mthe interval’s worse bound is inside M
--equivalent Mthe interval is inside ±M

--better, --noninferior and --equivalent need an interval, which rb gives from 3 units. --better and --noninferior also need to know which way is better, or rb refuses them: rb metric add top1 --maximize.

--min-n 3 sets the fewest units the criterion needs. With fewer, it is undetermined.

rb 0.8.4 has four more kinds of rule:

RuleReads
--count K --of Nwhether at least K of the N units meet the threshold. Undetermined unless the runs hold N units.
--best --of Nthe best unit’s own value, with --at-most or --at-least. It has no interval.
--relativethe relative change of the arms’ means, (subject - reference) / reference, in percent. The threshold is a percent too.
--means, with --source and --quotethe cited number as a difference of the arms’ means over all their units

Only a criterion with one of these rules is saved in format 2 (rb: 2). See The record’s format. Read a rule over seeds has worked examples.

A threshold comes from a file or from you, never from the agent:

  • A plan: --why "docs/plan.md:3". rb refuses the criterion if the line is missing, is a comment, or does not state the threshold.
  • A paper or other document: --source docs/baselines/resnet-paper.md --quote "| ResNet | 20 | 0.27M | 8.75 |" --locator "Table 6". rb checks that the file holds the quote and that it states the number.
  • Your words: --words "...", kept verbatim. Words an agent passes show as words (as given), since rb cannot know you said them.

--role primary, the default, is what the work claims. secondary and exploratory are context. A gate only steers where effort goes. A missed gate refutes nothing. It asks you to decide whether to park or continue the line of work.

Every finding says when its criterion was written, compared with its evidence:

rb saysWhen
declared before the evidenceit was written before its runs were produced
declared after the evidence existedsome or all of its runs already existed. rb says how many.
declared after the data was seenthe data it evaluates on was looked at first
order unknownsome evidence has no known time, such as numbers typed in with no --produced-at

Evidence that existed first still counts, but rb never reads it as if the criterion came first. Declare the criterion before the runs shows both timings side by side.

rb freeze <experiment> -m "before the runs" locks the experiment’s variants, settings and criteria, and records their hash. After that, a change fails with E_FROZEN unless the command adds --amend --why "...". rb keeps each amendment with who made it.

-q <question> links a criterion to a question, and only linked criteria count for that question. --when-met "..." keeps your words on what meeting the rule would mean. If the rule misses, rb does not claim the opposite.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0