Port provenance

At a glance

Tool

scripts/ci/check-port-provenance.py over the port_provenance package beside it

Inputs

The Ghidra dump corpus and the extracted overlay images, both gitignored

Gate

--check against scripts/ci/port-provenance-baseline.json, in the pre-commit hook; without the corpus it prints SKIPPED, NOT RATCHETED

The defect this exists for

The prize list's ink kernel carried PORT: FUN_801D5DE0 while live on both hosts. That routine is the casino prize list's row renderer: it indexes the prize table at 0x801E4518 and gates affordability on the coin bank, and the party's gold purse appears nowhere in its instructions. The mis-attribution came from a dump filename that names the image the routine was dumped from - an overlay that carries menu, save, shop and casino code alike - and it travelled: three subsystem pages described the one routine three ways, none aware of the others.

Why nothing noticed. Every text agreed with the disassembly about the operands. Only the subsystem label was invented. A checker that verifies cited addresses against the disassembly passes all three texts; the contradiction is only visible between them. The same law for load bases is dump-corpus integrity; this page applies it to function identity.

The signals

module-orphan reproduces the shop defect: the routine shares no rare table, no rare callee and no call edge with its module's siblings, while those siblings corroborate each other - and its rare prize table turns up under the prize-exchange module instead.

• Distinctiveness is the whole trick. The text writer is called from dozens of dumps, so "these two share a callee" is true of almost any pair. The checker counts, over every dump that printed instructions, how many routines form each address and call each target, and only rare features carry information.
• The cut follows the module. A module of window painters shares nothing rare by nature, so one fixed cut reports every painter as an orphan. The cut climbs per module until the module reads as cohesive, and only a member still unlinked there is a finding; a module that never coheres produces silence, not a finding per member.
• A pointer is not a claim. For dual-label, a directory row that links to the page it seems to contradict is filing that page's claim, and only subsystem, format and function-directory pages define a routine.

What was tried and deleted

A signal reporting every pair of tagged routines that shared a rare table across unrelated modules produced hundreds of findings, and none of the top ones was a defect: sharing a subsystem's own tables is the normal case, and a dispatcher module pairs with everything it dispatches. It survives only as corroboration on a module-orphan row. It is recorded because the premise is the attractive one, and the next person to build this instrument will reach for it first.

Precision, measured

The first run raised seven citation findings. Triaged by hand against the disassembly, six were checker defects and one was real. Among the defects: a tag line naming several routines was checked against every address on it, a dump's header size was read as its extent, and the indexed jump-table idiom lost its base register. Triage of the other signals found more of the same kind - calls to a sibling not counted as corroboration, a function-reference exemption keyed on whether a dump existed rather than on how the citation was written, and a tracker carrying an address through a pointer load and reporting a runtime field as a global the routine owns.

Where the code and the rows live

The package's rows.py holds what a contributor edits: modules split into directories for size (kept as one module so siblings and waiver keys survive the move), the pages that define a routine, and the accepted-site lists. A split-module row whose directory was renamed matches no file and silently regroups every tag under it, so path_check.py requires each row's directory and parent file to exist and exits 1 before the corpus is read.

Reading a report

python3 scripts/ci/check-port-provenance.py              # ranked report
python3 scripts/ci/check-port-provenance.py --live-only  # live ports only
python3 scripts/ci/check-port-provenance.py --addr 801d5de0
python3 scripts/ci/check-port-provenance.py \
    --funcs /path/to/checkout/ghidra/scripts/funcs     # from a worktree

Both corpora are gitignored; --funcs / --extracted point a worktree at another checkout's copies, because copying either in would stage Sony bytes. --live-only is the one to prefer: a wrong tag on an inert port is a doc bug, a wrong tag on a live one also mis-credits disc coverage.

The report is a worklist. --check fails when the unwaived count of any signal, or the count of waivers matching no finding, rises above the baseline - which sits at zero - so a commit that adds a finding owes it a read. Reviewed findings go in port-provenance-waivers.toml with a reason that says what was read and what it showed; "probably fine" is not a reason. dual-label, doc-citation and non-entry each found real defects at a rate worth gating on; module-orphan moves when a neighbouring instrument is corrected, so it is counted on the delta and not trusted on its own numbers.

Reading the tags against the bytes

The measurement the signals are ranked against is a person reading a tag's Rust beside the routine's disassembly. One full pass over the live tags in the engine's VM, world, audio, camera, locomotion and scene-host code sorted each into four verdicts:

The wrong addresses share a few shapes: a print at the wrong load base, a word that is not a function entry (a prologue word, an exit label), a module tag over code the module does not port, and the composite named for the leaf it inlines. non-entry reads the first two from verdicts already committed elsewhere (the ignore list, the dump-extent attribution, the overlay-band floor). An image-qualified tag (overlay_<label>_<NNNN>_<addr>) is checked from the bytes: it is accepted when that image, at its recorded base, opens a stack frame at the address. A second address named in passing on a tag line counts as a port of it, so spell it 0x<addr> or move it to a REF: line; a faithful copy of one fragment of a routine is a REF: naming the instruction range, not a PORT:.

A waiver can check the operands and miss the owner: a misbased print is byte-for-byte the real routine, so a review of what it does without asking which image holds it answers a question the signal did not ask.

A caller-site citation is checkable from the bytes

A row saying "called from the jal at 0x801E4B1C" cites evidence that lives in somebody else's body, so the dump corpus always read it as unsupported and each such row carried a waiver. The word at that address either is a call to the subject or it is not, so the checker now decodes the extracted images at their recorded bases and accepts six byte-derived relations: the citation reaches the subject, is the return address of a call to it, sits in an adjacent jr ra-delimited body, is co-sited with a citation that reaches it, holds it in a jump table, or sets up an argument a few words before a call to it.

• A waiver that matches no current finding is listed on every run and counted by the ratchet; it is a claim nobody can check.
• Dumps with no image tag were once left out of the evidence map, which silenced two signals over the whole executable while the run reported zero findings. One executable ships on the disc, so an untagged executable dump withholds nothing.
• Which body wins at an overlay VA is taken from the dump-extent attribution first, so adding one more dump under a different image cannot flip a verdict with no byte on the disc changing.
• A stale waiver is answered, silenced, re-keyed or moved; answered and silenced look identical from outside, and only a --show-waived run tells them apart.

Known-good shapes, excluded by construction

• One routine linked into two overlays: identical bodies at different VAs never produce a finding.
• VA aliasing: corroboration across the overlay band needs a shared image tag.
• Fragmentary dumps (a dispatcher printed as its head and epilogue) can corroborate a sibling and can never make a routine an orphan.
• Call targets decoded from a dump window at a wrong load base contribute no callee signal.
• Reusing a retail kernel in a different subsystem because the shape matches is not the defect; claiming the routine belongs to that subsystem is.

A differential is evidence

A retail kernel differential fails unless the routine at the declared entry ran off the named image and a Rust kernel computed what it wrote. The checker reads those declarations (no disc needed) and reports every tag site whose address a differential runs as verified; a module-orphan or citation finding there is settled by the run, not by a waiver. Provenance ranks every tag cheaply and gates on the delta; the differential proves the ones it runs - two rungs of the verification ladder.

An empty corpus is not a clean run

A fresh clone or a worktree has no dumps, and every port looks unimpeachable. A corpus-blind report run exits 2 and names the directory it looked in and the flag that fixes it; a run with dumps but no extracted images exits 2 as well, because it reports too much rather than nothing. --allow-vacuous accepts either. The gate's own --check mode prints SKIPPED, NOT RATCHETED and exits 0, the convention every corpus ratchet here follows, so the log still says nothing was compared.

The sibling question for the register

A settled thread graded disassembly may name only a function and a behaviour, and then reads exactly like a row checked against the listing. scripts/ci/check-register-address-cites.py lists every such row with no instruction-shaped address in its answer or its write-up. It is a worklist and always exits zero: many listed rows are not code claims at all and want a different grade rather than an address. A cited row is not thereby a correct one; the tool answers whether a claim can be re-read.