Disc coverage
How much of the game's own bytes the project can account for, measured against the disc rather than against the project's own notes. This is where the byte-denominated figures on the landing page come from - and the one instrument that can say how much of the game is left.
At a glance
- Tool
scripts/ci/disc-coverage.py; attribution sweepscripts/ghidra-analysis/attribute-dump-extents.py- Two halves
- Code is byte-exact (a byte is inside a dumped function or it is not); data is format recognition, an upper bound
- Inputs
- Ghidra dump corpus (
ghidra/scripts/funcs/) +extracted/PROT/categorize.json- both gitignored, disc-derived - Gate
--checkratchets againstscripts/ci/disc-coverage-baseline.json; skips and passes without disc data- By-product
- The largest un-dumped code runs, listed as a dump worklist derived from the bytes
- Sibling
- Port catalog - the same questions denominated in what the project cites
What this solves
The port catalog tracks dumped, documented and ported over the set of addresses this project cites - the right instrument for steering work, but blind to a subsystem nobody has cited. disc-coverage.py takes the denominator from the disc instead. Reach for it when you want to know what code nobody has looked at yet, or when you want a progress figure that cannot flatter itself.
The distinction runs in both directions: closing a byte-denominated gap widens the citation graph, because a newly dumped and documented function becomes a row on the port worklist. A rising port worklist after a dump pass is the measurement getting wider, not the work going backwards.
- Dump headerseach Ghidra dump names its entry VA + byte length - a real interval
dump_header.py - Merge + subtractmerged intervals subtracted from each image's extent leave the un-dumped remainder
disc-coverage.py - Classify gapscode vs data, then by shape - only one of six shapes is work
- Attribute overlaysshared load addresses: credit an image only where its bytes hold the dump
dump-extent-attribution.csv - Worklistthe biggest code gaps, ranked, become the dump worklist
Quick start
| Command | Does |
|---|---|
python3 scripts/ci/disc-coverage.py | Report -> target/disc-coverage/ |
python3 scripts/ci/disc-coverage.py --md | Markdown to stdout as well |
python3 scripts/ci/disc-coverage.py --check | Ratchet against the baseline |
python3 scripts/ci/disc-coverage.py --update-baseline | Rewrite the baseline (needs a reason in the commit) |
The data half reads extracted/PROT/categorize.json. That file is a cache written by asset categorize and never regenerated automatically, so a tree whose detectors have moved on keeps reporting a stale classification - and the gate passes either way. Regenerate before trusting a data figure or taking a baseline:
./target/release/asset categorize extracted/PROTThe two halves measure different things
Three instruments share the work, each with its own denominator: this page's code half (is a byte inside a dumped function?) and data half (is an entry's format class known?), byte accounting under the data half (which bytes does a parser consume?), and the port catalog, which measures the addresses the project cites rather than the disc.
| Kind | What a percentage means | |
|---|---|---|
| Code | byte-exact | a byte is inside a dumped function, or it is not |
| Data | format recognition | an entry's format class is known; its bytes are not individually accounted for |
The data figure is an upper bound: knowing an entry is a scene_vab_stream is not the same as consuming every byte inside it. That gap has its own measurement, a third denominator rather than a refinement of this one: byte accounting reports, per entry, which bytes a parser actually claims, and scripts/ci/byte-account-coverage.py --check ratchets it the way this page's gate ratchets the code figure. The data denominator also counts some disc bytes more than once - entry weights use max(indexed_size_sectors, footprint_sectors), and only the footprint is an entry's real extent (see PROT TOC) - so the totals run to roughly 2.5× the archive's real size. Quote the shares, not the totals; when a single entry's weight makes a residue look significant, check its footprint first.
A statistical class is not a verdict
asset categorize ends in a statistical fallback that buckets unclaimed entries by zero fraction and entropy (mostly_zeros, unknown_low_entropy, ...). When reading the per-class table:
- A statistical class name is the absence of a finding, never a finding -
mostly_zerosmeans "sparse and unclaimed", not "empty". - The placeholder column is only as trustworthy as the detectors that ran before the fallback; a format with no detector is invisible to it.
- Entries of an identical, exact size are the cheapest lead: a size repeating across a hundred entries is a fixed-layout format whatever its statistics look like.
- A statistical class can swallow content by dilution - a whole-buffer printable-ASCII ratio misses an image that is mostly bss with its literals in the first sector. Structure at a known offset beats a whole-buffer ratio.
A statistical class is an inference, not a format: every scene's sparse 0x12000-byte field map once scattered across the statistical classes, mostly into "explained placeholder", until a structural detector claimed them.
How code coverage is computed
Every Ghidra dump header carries an entry address and a byte length (== FUN_800402f4 ... size=7904 bytes, 1976 instructions ==), so the dumped functions are real intervals over an image's address space. The script merges them, subtracts them from the image's extent, and splits the remainder into code and data - a PS-X EXE's text segment carries read-only data in the same span, and counting string tables as "un-decompiled code" would understate coverage badly. Each gap is classified statistically (share of plausible MIPS opcodes, density of 0x80xxxxxx pointer words), checked against a known-code control region.
A code gap is not automatically un-analysed code
The report classifies each code gap by shape, and only one of them is work:
| Shape | What it is | In the code denominator |
|---|---|---|
code | genuinely un-dumped instructions - the work | yes |
data | the opcode statistic rejects it: rodata in the text segment | no |
padding | every word is nop: inter-function alignment | no |
mostly_padding | at least half the words are zero | no |
no_exit | 1024 bytes or more with no jr ra in them | if the statistic passes it |
return_tail | a jr ra (+ nop) the preceding routine's analysed body stops short of | if the statistic passes it |
bios_thunk_slot | the delay slot of a jr $t2 PSX BIOS-call thunk | yes (tiny-gap fiat) |
psyq_lib_stamp | an 8-byte librarian version stamp between link modules | yes (tiny-gap fiat) |
constant_table | one repeated non-nop constant: a data table resident in the text segment | yes (tiny-gap fiat) |
The end-of-body and linker shapes persist however much is dumped, which is why the figure asymptotes short of 100%. Most shapes are reported, not subtracted - the denominator keeps them, so the ratcheted figure stays comparable across classifier changes. The padding shapes are the exception, excluded by a structural rule rather than by the opcode statistic: a word of zeros decodes to nop, so a majority-zero run passes the code test outright and would otherwise sit at the top of the worklist while the census called it padding.
The same encoding defeats the attribution sweep, one level up. That sweep names the image an ambiguous extent belongs to by asking which image's own bytes reproduce the dump's opening instruction window at that address - and a window of nops reproduces inside any image's zero fill, at any base. A window's usable length is therefore its non-nop instruction count. Two shapes of false credit came out of the corpus when that guard was added: an all-nop window read as byte-identical across two images that agreed about nothing but being empty, and a 24-instruction window carrying one lone non-nop word in a run of zeros read as a unique attribution, crediting one image with a 20 KB extent that is over nine-tenths zero fill. Removing the credit moved most images' floors up - extents the zero windows had made ambiguous for several images stopped being ambiguous at all - and only the images the zero fill had been credited to down.
What the executable's gap turned out to be
Working the SCUS_942.54 gap list until it stopped yielding produced on the order of a hundred function entries. A five-form reference sweep (address-reference scan) over them splits three ways:
| Share | Why it had no dump | |
|---|---|---|
| no reference of any form | ~3/4 | Ghidra creates functions from the call graph; a routine nothing references gets no function record, no dump, no citation - invisible to any citation-denominated instrument. |
referenced from SCUS_942.54 | ~1/5 | mostly reached through the entry-stub / init path, or from a body whose own analysis was incomplete. |
| referenced only from an overlay | a few | live code whose caller lives in an image the SCUS-only analysis never sees - "zero static callers is not dead", showing up as a coverage gap. |
The first row is the point: the bytes find the class the citation graph structurally cannot. Caution: "no reference exists" is a statement about static references, and much of that band is linked-in library material the game does not call.
Dump exclusions: a census, not a count
The dump corpus spells every header field more than one way, so the report names each reject class and separates two populations: answers (pointer_stub, nofunc_record, data_window, not_a_dump - the corpus recording a result, not defective and not work) and defects (zero_insns, gapped_stream, no_extent, empty_dump - a dump that cannot evidence its own extent). A pointer stub is the corpus doing the right thing with a mid-function address; zero_insns is Ghidra's "bad instruction data" - a data window asked for as code, excluded rather than credited.
Reading the header looked like the trivial part and was the part that went wrong: a dozen dump scripts each spell the header differently, and each instrument once grew its own regex, silently rejecting a different set of real dumps and reporting them as a corpus deficiency. Sharing one parser (scripts/ghidra-analysis/dump_header.py) across every instrument is the fix; the general lesson is on dump-corpus-integrity.
Why overlay rows can read "not meaningful"
SCUS_942.54 is the only image with an unambiguous answer: one load image, one fixed base. Several overlays load at the same base (0x801CE818), so a dump whose entry lands in that band cannot be attributed to one image by address alone. Rather than publish a number that quietly double-counts, each overlay row carries the share of its covered bytes that could not be placed; above 50% the figure is replaced by not meaningful and the row leaves the ratchet baseline. That is also why the landing page's decompilation bar quotes the executable alone.
The ambiguity is resolved where the bytes can resolve it: attribute-dump-extents.py disassembles each extracted image at its static-overlays.toml base and asks which images actually hold a dump's bytes at the VA it prints. Its per-extent verdict is committed as scripts/ghidra-analysis/dump-extent-attribution.csv:
| Verdict | Meaning | What the gate does |
|---|---|---|
unique | one image holds those bytes there | credit only that image |
identical | several hold byte-identical code there | credit each of them |
misbased | the bytes live at another VA entirely | credit nobody |
gapped / data | not a coherent function body at that VA | credit nobody |
short / unresolved / no_disassembly | the window cannot sign it | residue: stays ambiguous |
The key is the extent (entry, bytes), not the dump filename, so the file does not rot when a dump lands or is renamed. Attribution is optional: without the CSV every overlay extent stays ambiguous - the honest upper bound, just a looser one.
What the residue is, and which count to quote
| Residue shape | What would close it |
|---|---|
| a few-instruction window no image reproduces at that VA | nothing cheap - too short to search for elsewhere without inviting a coincidental hit |
| bytes in no extracted image at any VA | an extraction, not a dump - live-RAM captures of overlays never statically extracted; the static overlay pipeline's job |
| two dumps at one extent resolving to different images | already answered: several routines share the range - a finding, not a gap |
The sweep asks two questions with opposite sensitivities to window length - does this image reproduce the window at this VA (fixed-offset test, three-instruction floor) and do these bytes appear anywhere (a search over millions of positions, eight-instruction floor); --validate-short-floor sets the relaxation from its own control. The code table carries three counts, and the unit matters: dumps is per dump file; VA-ambiguous is the share of an image's covered bytes the attribution could not place, which is the unit the coverage figure itself is stated in; by extent is the same question counted in whole extents. Reading the extent count as the discount on a byte figure withheld an upper bound from most of the slot-B band over fractions of a percent of its bytes - the corpus carries a long tail of 4-to-36-byte Ghidra fragments, and an extent count weighs each of those like a 6 KB module. Neither is per dump file: one extent can back dozens, and weighting by how often the same bytes were dumped measures the corpus, not the image.
"Unresolvable by address" is a statement about one method, not about an image: byte attribution still places most extents of a nested overlay span. Superseded readings are on do-not-re-walk.
Gate behaviour
- Both inputs are gitignored, so a clone without disc data exits 0 and reports SKIPPED - the same skip-and-pass convention as the
LEGAIA_DISC_BINtests. The ratchet only has teeth on a machine with the disc. --checkcompares against the committed baseline; coverage may only go up, within half a percentage point. A baseline moving down is a claim that needs a reason in the commit message (--update-baseline).- The report lists the largest un-dumped code runs in
SCUS_942.54by size - the one dump worklist the citation graph structurally cannot produce. - The landing-page figures are refreshed by
scripts/ci/update-progress-metrics.pyand committed, because the site is built where the disc is not. - The homepage's third row, how much of the port is checked against retail's own code, comes from the same JSON's
verificationblock: the share of// PORT:-tagged addresses a retail kernel differential runs. It is counted from the tests' declarations, so it refreshes without the disc and the freshness check recomputes it exactly (the verification ladder).
Per-image port status
The homepage's whole-project tiles saturate at 100% once the worklist is empty. The per-image table below them asks the question that still moves: for each runtime code image - the executable and every overlay - how much of that image's code is ported. Once every game image is complete the cards fold behind one line, as the saturated tiles collapse to one checked line each; a single image or track falling short brings the full view back.
- Functions are the entries this instrument's byte attribution places in the image, with the inherited tail cut. Ignore-list rows that say "no routine begins here" (labels, interior fragments, phantom prints) are not functions and leave the count.
- Ported carries a
// PORT:tag. Native is an ignore-list scope claim: PsyQ, BIOS, libgte, GPU and CD plumbing the engine replaces wholesale, or a routine with no retail behaviour. A VA-aliasing exemption excuses an address, not any one image's routine, so per image it excuses nothing. - Code identified is the share of the image's code bytes inside a placed function. A function list cannot see a routine nobody dumped, so every card prints it beside the percentage.
- Fully ported means nothing open, no un-dumped code, and at least 99% of the code bytes identified. The 64 slot-B cast modules (PROT 0903-0966) render as one band with a modules-complete count.
The table is committed JSON refreshed with the tiles; the freshness check recomputes digests of its committed inputs (port tags, ignore list, attribution, overlay map) and warns when they move. Full method: disc-coverage.md.