Skip to content

The record's format

Every file in .rb/ starts with its format, rb: 1 or rb: 2, then its kind. One JSON Schema (draft 2020-12) covers both formats. It is served at its $id, rabbitbrain.ai/schema/rb-v2.schema.json, and kept in the repository at schema/rb-v2.schema.json.

Terminal window
# the whole format, for any JSON Schema validator
rb schema all > rb-v2.schema.json
# one kind: project, question, experiment, metric, criterion, run, analysis, resource or decision
rb schema run
# a finding's shape: computed on every read, never stored
rb schema finding
# what every command prints with --json
rb schema envelope
# read one record file and check it against the format
rb show .rb/experiments/int8-ptq/runs/<id>.yaml

The files use a strict subset of YAML: mappings, lists, text, numbers, true, false, null and comments. rb refuses unquoted values that YAML 1.1 and 1.2 read differently, such as yes, no, on, off, dates, numbers with leading zeros, and 1e-4 without a dot.

rb fmt rewrites files in canonical form and keeps comments.

Format 1 does not change. A newer rb that adds a field writes that file in format 2, and an older rb says to upgrade (E_STATE_NEWER) instead of reading part of it.

A criterion is the rule a result must meet. Format 2 adds four kinds of rule to criteria, and nothing else:

  • a count: over: count, with count and of
  • a best of N: over: best, with of
  • a relative change: measure: relative
  • a cited difference of means: measure: means

Criteria explains each one.

A file with rb: 1 and one of the four, or rb: 2 and none, is refused (E_STATE_CORRUPT).

rb export --with-record writes record_format: 2 in the bundle’s manifest when a criterion’s file is format 2, and 1 otherwise.

What each version does:

  • rb 0.8.0 to 0.8.4 read both formats. rb 0.8.4 writes format 2 only for a criterion that uses one of the four, retracted or not.
  • rb 0.7.0 and 0.7.1 read format 1 only. Where a record holds a format-2 file, they read the other files but say to upgrade, fail every --fail-on gate and write nothing to the record. They refuse to clone a bundle that holds one.

Upgrade everyone who shares a record together.

Terminal window
$ rb schema ir
rb's vocabulary: every record kind, what it means, the command that writes it, and the sources rb requires.
question what the work is trying to find out
a tree: a sub-question names its parent, and an experiment may be filed under several; where it stands is read from what is below it, never a child's negative as its own
write: rb question add "<what are you trying to find out?>" --parent <q>
provenance: wording as asked
experiment what is tested
its purpose or hypothesis, one baseline and variants with roles, varies naming the intended differences; planned until its first run; an undeclared difference between variants is a confound
write: rb experiment add "<title>" --hypothesis "<h>" --baseline <b> --candidate <c> --varies <s>
provenance: declared, then checked against its settings
variant one arm of an experiment
baseline, candidate, control or ablation; its own settings override the shared ones
write: rb variant add <exp> <name> --role <r>
provenance: role declared
setting what the experiment changes or depends on
a value with where it comes from; population.* declares the counting structure and never blocks
write: rb spec set <exp> <name> <value> --source file#key
provenance: REQUIRED: a source rb can read, or --unknown, or a vouch (rb decide <exp>/<name> accept)
criterion what would count on one comparison
a rule over one comparison (subject, reference, metric, estimand), with a role (primary, secondary, exploratory, gate), declared by someone at a time before the evidence it reads; explicit question links may keep the author's verbatim interpretation when met, never an answer inferred by rb
write: rb criterion add "<c>" -e <exp> -q <question> --metric <m> --subject <a> --reference <b> --better
provenance: declared by whoever wrote it, with the user's words when an agent gives them; question links snapshot the question and retain author, time, reason, session and revisions; REQUIRED for cited numbers: --source --quote --locator, checked against the file
run one execution, and what it produced
results with a receipt: actor, chat session, the agent (its name, version, provider and model), time, repository state, command, environment, the resources read and made, when its output existed; the same output recorded twice is one run; a repeat checks a run and adds no unit; a wrong run is retracted (whole, or --scope for named measurements), never deleted
write: rb run add <exp> --from <file> --set seed=N --input <resource> --config <resolved>
provenance: REQUIRED: the receipt is taken when it is recorded; --from and --config are read and hashed; typed numbers are marked typed
analysis a computation over saved outputs
the comparisons it produced, each with its estimate and interval, the unit it resampled, and where it came from; a reanalysis is an analysis, never a run; a later one supersedes it with a reason, and its readings stay as history
write: rb analysis add <exp> --compare "a/b:m=est[lo,hi]" --method "<m>" --unit "<u>" --units N --source <file>
provenance: REQUIRED: computed from an output file rb hashed, or reported from a document rb checked, or marked typed
comparison subject against reference on a metric
the unit of state: an estimate, its interval when there is one, what it resampled; changing the metric, the reference or the estimand is a different comparison
write: read by a criterion; produced by an analysis or by rb from the runs
provenance: an analysis, or rb's reading of the runs
result a number a run reports
named <variant>.<metric>, change.<metric> (candidate minus baseline) or plain; compared across variants in rb compare
write: NAME=VALUE on rb run add, or --from <file>
provenance: the run's receipt
resource a dataset, checkpoint, code, environment or artifact
identity (name, version, hash), locations (any provider), and how to get it again; availability is computed: present, changed, remote, reconstructible or lost; a run records what it found when it read or made one
write: rb resource add <id> --type dataset --name "<n>" --url <u>
provenance: declared; a local file is hashed when recorded
metric what is measured, by name
unit, direction, aliases; a result naming an alias is read through it
write: rb metric add <m> --unit <u> --minimize
provenance: declared
finding a criterion read on axes, computed
result, rule, timing, units, integrity, review and history, computed from the runs and analyses on every read, never stored; no single word stands for it
write: read it: rb show <criterion>
provenance: derived
decision a call, with what it was made on
accept, reject or investigate a finding, an experiment, a question or a setting; continue, park, stop, pivot or ship a line of work; read against the readings it was made on, with when to revisit; one recorded in an agent session is proposed until the user confirms it (rb confirm), and a past one (--past) is Retrospective
write: rb decide <subject> <action> -m "..." --revisit "..."
provenance: REQUIRED: who made it and why, with the chat session when known and the user's words verbatim when an agent acts on them; for a past call, when it was made and the document that states it
note words about an experiment or the project
an assumption, a caveat, context; kept verbatim with who wrote it; changes no finding
write: rb note add "<text>" -e <exp>
provenance: text as written
relation how one experiment stands to another
reproduces, extends, follows_from or compares_with an experiment here, a uid, or a published manifest; structure, not evidence
write: rb experiment relate <exp> reproduces <target>
provenance: declared

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0