Ratchet integrity
Most of this project's checks are ratchets: a measurement is compared with a number committed beside the code, and the check fails when the measurement gets worse. A ratchet can be undone two ways without the ratchet itself noticing. Someone edits the committed number, or the measurement quietly stops running. Each has its own gate, and neither gate needs the game disc.
At a glance
A baseline may not loosen silently
Every ratchet accepts a new baseline in the same commit as the code that moved it. Most of them read inputs a build runner does not have (the disc, the function dumps, the save-state library), so in automated runs they report SKIPPED and pass. What is left on the runner is the baseline file itself, which is plain text, and anyone can lower a number in a text file.
scripts/ci/check-baseline-loosening.py reads each baseline at a base revision and in the working tree and walks its numeric values. A value that moved the wrong way is a loosening:
| Rule | What counts as a loosening |
|---|---|
up | The value fell: a score, a coverage figure, a ladder tier |
down | The value rose: a worklist, a count of unparsed bytes |
pin | The value changed at all: a setting of the measurement |
set | An entry was added to a list of known-bad items |
| any | A baselined key, or the whole file, disappeared |
A table in the script says which rule each key takes. A numeric key that no rule covers is itself a finding, so a new baselined figure cannot arrive unclassified.
A loosening fails unless the ledger scripts/ci/baseline-loosenings.toml records that exact from -> to step with a reason:
scripts/ci/check-baseline-loosening.py --propose >> scripts/ci/baseline-loosenings.toml
# then write each `reason`An entry matches one transition only, so it cannot excuse a later fall, and it is never removed. The ledger is the history of what the ratchets gave back, which a squash-merged log would otherwise bury.
The pre-commit hook compares the working tree with the merge-base against main. The automated gates job compares a pull request's merge commit with the pull request's base, and a push with the commit it replaced.
A skipped test is not a passed test
A disc-gated test that finds its input missing prints a skip line and returns, and the test runner reports ok. On a machine with no disc the whole gated suite is green. On a machine with the disc, a test whose second input is gone (the extraction folder, the save library, a memory-card file) is green too.
scripts/ci/run-disc-tests.sh runs the workspace tests with every test's output kept, and scripts/ci/disc-test-tally.py reads the log:
- A test that skipped for want of the disc fails the run. With the image present, such a line means a gate that is reading the wrong thing.
- A test that skipped on another input is counted, grouped by the line it printed, and written to the job summary.
LEGAIA_MAX_OTHER_SKIPSturns the count into a bound.
The tally depends on the skip line. A test that returns early without printing one is invisible to it, so a gated test prints [skip] <what was missing> before it returns and names every input it lacked. Only LEGAIA_DISC_BIN itself is "the disc"; another disc's variable is another input.
The suite runs through this script in a manually dispatched job (or one a disc-test label on a pull request starts) and on a nightly schedule, which skips a night when main has not moved since the last run that passed. The log holds output derived from the disc and stays on the runner.