Recomp differential oracle

Moving parts

Captured traces are Sony-derived (retail game RAM values) and must stay untracked - like every capture artifact in this repo, they live under a scratch directory, never in git. The synthetic fixtures in scripts/recomp/test_trace_diff.py are the only committed trace-shaped data. The recomp workspace itself is a separate untracked tree; nothing from it is committed here.

The recomp side

The recomp runtime exposes a JSON-over-newline TCP debug server. probe.py wraps it as a library (RecompClient) and a small CLI:

python3 scripts/recomp/probe.py --port 4494 ping
python3 scripts/recomp/probe.py --port 4494 read 0x8007050C 8
python3 scripts/recomp/probe.py --port 4494 press cross 30
python3 scripts/recomp/probe.py --port 4494 load-state 4 --expect-mode 0x15
python3 scripts/recomp/probe.py --port 4494 launch --cache-dir /tmp/mycache --wait-tcp

Environment contract: LEGAIA_RECOMP_DIR points at the recomp workspace, LEGAIA_RECOMP_BIOS at a PSX BIOS image, LEGAIA_RECOMP_PORT sets the default port; all three are overridable per-invocation. Headless instances (--headless --debug-port N) serve TCP in tens of seconds and can screenshot with no X server - the display disabled error some calls return means the guest GPU's display-enable bit is off (early boot, between attract segments), not that the host lacks a display.

The savestate-resume fix

Stock, the recomp runtime cannot resume a savestate: its loader forces cpu->pc back to the game entry, which is the BSS-clear routine, so every load faithfully restores RAM and then immediately wipes it - and the ack is {"ok":true} either way. One line fixes it (honour the recorded resume PC), applied by scripts/recomp/apply_boot_state_fix.py - a script rather than a .patch so no third-party source lines are vendored under an incompatible license, and so it survives upstream line drift. Rebuild the psx-runtime target, not the executable-named target (make treats the existing binary as satisfied and silently does nothing).

Three faults all present as "the trace was taken at the boot entry": a self-wiping runtime, a stale build, or a snapshot that predates resume-PC capture. scripts/recomp/preflight.py separates them and is wired into probe.py launch, trace_capture.py --savestate, and failed --expect-mode / --expect-scene checks, so a genuine divergence is not written off as a harness problem.

Protocol traps the client bakes in

• One request per connection - the server closes the TCP connection after every response; a persistent-socket client dies on its second request.
• press carries a raw active-low SIO pad word in a field named buttons (a wrong key is silently ignored), and returns before the hold elapses; RecompClient.press sleeps the hold out. Use ≥ 30-frame holds for confirms.
• Savestate loads are staged and silently fallible. The load executes at the next block boundary, dropping every connection; a stale slot loads "successfully" into the wrong state. Always pass --expect-scene / --expect-mode; the client retries verification reads through the reconnect.
• Synchronous stepping is removed. The frame-exact primitive is the per-frame snapshot ring (set_snapshot, 4 slots × 128 bytes, + read_frame_ram to read a region as of a specific frame); trace_capture.py's ring engine is built on it.
• vram_peek clamps to 128 px per call (the client chunks); dirty_exec_hot PCs are KSEG-masked physical - OR 0x80000000 before resolving against overlay VAs.
• Kill by PID, never by pattern (pkill -f matches your own shell), and one port, one instance - launch refuses a port that already answers.

Canonical trace shape + channels

One JSON object per line; every field except frame is optional per line (absent = not captured that frame):

{"frame": 32915, "scene": "jou ene", "mode": 21,
 "cam": {"pitch": 32, "yaw": 2408, "roll": 0, "h": 256,
         "eye": [0, 1280, 7920], "focus": [0, 0, 0]},
 "player": {"x": 100, "z": 200, "heading": 0},
 "actors": [{"i": 1, "x": -40, "z": 80, "heading": 1024}]}

Units are retail on both sides: angles in the PSX 12-bit space (4096 = full turn), positions in retail world units, mode in the retail game-mode word space. The recomp-side address map reads the pinned retail globals (camera rotation trio, projection H, eye + focus trios, player pointer, scene name, game mode - see the memory map).

cam.eye and cam.focus are not a world-space eye / look-at pair. They are the retail camera globals verbatim: cam.eye is the post-rotation eye-space translation trio (its Z is the eye-back depth, not a world position), and cam.focus stores the focus negated in X and Z. An engine side that emits world-space (eye, look_at) here compares different quantities in different coordinate frames - a total, permanent divergence on both channels that looks exactly like a camera defect and cannot be closed by any change to the camera. Check the convention before reading a camera divergence as a camera bug.

Workflow

1. Capture a recomp trace

python3 scripts/recomp/trace_capture.py --port 4494 \
    --frames 100 --map camera --out /tmp/scratch/recomp_cam.jsonl

Built-in maps: camera, player, scene; --actors adds a configurable per-actor sweep. Two capture engines: ring (frame-exact, bounded by the 4×128-byte snapshot budget - the camera map alone uses all four regions) and poll (best-effort live loop tagged with the frame counter; required for pointer-chasing actor sweeps). The instance must already be in the scene of interest: prefer loading a parked savestate (seconds), verifying scene + mode + an advancing frame counter after every load; the fallback is a cold boot that loads a memory-card save through the title screen, where the traps are that CONTINUE is not the default cursor position and the load confirm defaults to No - screenshot each screen rather than pressing blind.

2. Emit the engine trace

legaia-engine sim-trace --scene town01 --disc "$LEGAIA_DISC_BIN" \
    --frames 100 --out /tmp/scratch/engine_town01.jsonl

Boots a BootSession on the scene, drops into a live field scene (enter_field_live - field VM, locomotion, and camera events armed the way the windowed host arms them), ticks the frames, and emits the same canonical records. Every cam.* channel is read from the engine's live retail camera globals - the same ten words the recomp-side map reads - so the two sides report the same quantity in the same frame. Per-tick emission of existing sim state only; the subcommand adds no simulation features. mode maps SceneMode onto the retail game-mode word and is omitted for modes with no retail equivalent, so a diff flags them as absent rather than faking a match.

3. Diff

python3 scripts/recomp/trace_diff.py /tmp/scratch/recomp_cam.jsonl \
    /tmp/scratch/engine_town01.jsonl --tol-angle 2 --tol-pos 2

Alignment defaults to the first camera change in each trace (the two sides boot from arbitrary frame counters, but the first scripted camera cut is the same event on both); --offset N overrides. Two auto-alignment edges worth checking before believing a report: sim-trace's first record is a pre-tick boot sample (--skip-lead-b, default 1, drops it), and a reported first divergence can simply be the first frame of the overlap - if the context window has no rows above it, nothing was compared before it.

Comparison is per-channel over the aligned overlap (cam.yaw, player.x, actors[2].heading, scene, mode, …). Angle channels reduce both values mod 4096 before measuring wraparound distance (without the reduction the wraparound term goes negative and the channel reports OK on every frame); position channels use absolute distance; scene / mode compare exactly. Channels present on only one side are reported and skipped, not counted as divergence. Each divergent channel reports its first divergent frame with a ±5-frame context window; the exit status is non-zero when anything diverged. scripts/recomp/test_trace_diff.py (pure python, synthetic fixtures) locks the alignment + wraparound + tolerance semantics.

What the state oracle surfaced

Run against the opening-chain scenes, the strongest evidence needs no alignment at all - comparing each channel's range over a window sidesteps the offset question entirely.

• The camera-position divergence had two stacked causes, and the outer one was a measurement artifact. The engine originally emitted world-space eye / look_at on channels whose retail side is the eye-space translation trio and the negated focus globals (above) - disjoint value sets by construction. Underneath sat a real defect: Camera modelled no eye-space translation trio at all, so scripted shots rendered from the follow orbit. Both are closed: Camera now carries the ten retail globals, applies each masked slot per-axis, and runs camera_mover for apply != 0 beats; the angle channels reproduce retail exactly over the opening chain and the eye channels overlap where they previously could not.
• Pose changes were held, not interpolated - same root cause, closed with it: windows that carried 2-6 distinct poses now carry hundreds.
• The remaining position gap is a timeline-phase difference, not a pose error. sim-trace boots a scene cold at its frame 0; a retail capture of the same scene arrives through the prologue chain already several beats in, so an aligned frame compares the engine's beat 0 against retail's beat 5. The same confound covers scenes where the cold-booted engine runs no camera script at all - a camera beat gated on arrival story flags produces exactly that reading with no camera-system defect. Distinguishing the two needs the engine driven through the chain, not booted into the scene.

Note-level audio differential

The same alignment-and-first-divergence shape, applied to BGM: instead of camera channels per frame, it compares note sequences - the stream of key-ons a sequencer asked the SPU for - which is the level at which "missing notes" is a measurable claim rather than an impression. audio_note_capture.py records the retail timeline from the recomp's SPU rings, the engine-audio note-trace binary plays the same track through the engine's own sequencer, and note_diff.py aligns on note ordinal (captures start mid-track; the frame counters have unrelated origins). Both sides snapshot the same key-on instant - ADPCM start address, pitch, per-voice volumes, raw ADSR words - so a divergence localises directly: a missing note-on means the sequencer never asked, a wrong start address means tone selection diverged, a wrong pitch means the note or its bend resolved differently.

python3 scripts/recomp/audio_note_capture.py --port 4472 \
    --seconds 30 --out /tmp/scratch/recomp_notes.jsonl --summary
./target/release/note-trace --extracted extracted --track 0 \
    --frames 1800 --out /tmp/scratch/engine_notes.jsonl
python3 scripts/recomp/note_diff.py /tmp/scratch/recomp_notes.jsonl \
    /tmp/scratch/engine_notes.jsonl

• The unclocked-SPU trap. A --headless recomp never clocks the SPU (the host audio pump drives it), so every voice sits frozen at envelope 0, the driver believes all 24 voices are free forever, and nearly everything keys onto voice 0 - a plausible-looking capture that is entirely an artifact. The capture script refuses to run unless SPU frames advance; get a clocked instance with SDL_AUDIODRIVER=dummy xvfb-run -a <runtime>, and verify yourself that SPU frames advance at exactly 735.0 per guest frame - the guard only checks advancing, not the rate.
• Raw SPU addresses never match (each side lays out the bank itself), so note_diff.py maps addresses to dense VAG ids by ascending order within each trace. The renumbering is per-trace: a tone played on one side only shifts every id above it, so when the two sides' distinct-VAG counts differ, pitch is the reliable channel.
• Matching the track: every scene that starts BGM selects a global-pool id, so every capturable track is a music_01 entry note-trace --track can reproduce (see audio). Read the live BGM id out of the running recomp rather than guessing, and cross-check the track number against note-trace --list's prot_entry column - --track is not the sound-test slot.

Divergences a matched run surfaced - both closed

• Per-voice volume was written in the wrong domain. The engine's key-on volumes occupied 0..127 where retail's occupy the SPU's 14-bit 0..0x3FFF, leaving every key-on short by a factor of 0x81 and diverging the vol channel at the first note of any track. fire now carries the retail chain of FUN_80067550 with its staged truncation points (vel × bank_mvol × 0x3FFF / 0x3F01, then × prog_mvol × tone_vol / 0x3F01, program-level ProgAtr factors included), and the sequencer path's closing square taper (v²/0x3FFF per side) lands in sequencer.rs's channel_mix.
• Tone selection collapsed. On banks with many tones the engine keyed only a few distinct VAGs while retail draws across the whole table. Root cause: the VAB file packs one tone page per used program, and the engine indexed the packed pages with the raw program number, where retail resolves a program to its page by rank among the used ProgAtr slots (FUN_80068d94 writes the rank table at VAB open, FUN_80068b98 reads it at program change). A large share of the disc banks author sparse program sets, so raw-number indexing landed on the wrong page or fell off the table outright. VabBank::upload now expands the pages into program-number space; the law is pinned corpus-wide by engine-audio/tests/real_vab_program_mapping.rs. A wrong tone carries a wrong base note, so read vag first and treat pitch as downstream of it.
• The v (voice index) channel differs whenever either of the above does - allocation order is a function of the note stream, an effect, not an independent finding.

See also

Determinism + replay is the engine-vs-itself side of the parity stack; this page is the engine-vs-retail-recomp side. The emulator harnesses cover breakpoint / save-state observation of retail; the recomp path adds cheap frame-tagged structural reads at full speed. Full reference incl. the complete trap catalogue, address maps, and route-2 navigation: docs/tooling/recomp-differential.md.