Watch runs and review

How swarmchase watch collects and reports, and how swarmchase review labels what it finds. The command summaries are in the CLI README; the day-to-day flow is in the admin workflow.

The data directory#

Every watch, review and case command works in one data directory: --data (-d; the watch also takes --out and -o), else SWARMCHASE_DATA, else data/ relative to where the command runs. On the runner it is $SWARMCHASE_DATA outside the checkout (runner). It is gitignored and never committed. A command that reads existing data (review, case, notice add) prints a hint naming the directory it resolved when that directory holds no watch data, labels or cases. A watch run refuses such a directory unless given --init, so an unset or mistyped SWARMCHASE_DATA never starts a fresh store beside the real one. Start a data directory once with swarmchase watch --init; on the runner, run the watch workflow by hand with its init input (runner).

Collection#

watch is incremental. It pulls the sources in watchlist.toml, each of which carries a rationale: the agent behaviour it catches, the rule it feeds and what its noise looks like.

• Cursors. Each source keeps a time cursor in state.json. New records are deduplicated by ID and appended to the source's store, each to the file of its capture month. A record stamped in the future moves the cursor no further than the run's own time. A source that cannot page and returns limit records moves its cursor no further than the oldest of them, since what lies before may be missing. IDs of records with no time stay in the seen list, so they are never added twice.
• Wayback paging. Wayback's index is sorted by URL rather than time, so a pull cut short by limit, or by a failed page, resumes from its resume key on the next run. A host too big for one index query is read by path prefix, and slice_days reads the time range in windows.
• Checkpoints. GH Archive, Wikimedia EventStreams, Hacker News, Bluesky, GitHub search and CT logs read on from where the last run stopped (mark in state.json), within a per-run budget. Every page read counts as progress, matches or not, so a failure after matchless hours resumes after them. A GH Archive hour whose file is corrupt or cut off is skipped and counted, not retried each run.
• Unavailable hosts. A host answering HTTP 503 gets ten minutes of retries; if it stays down, later requests to it in the same run are not sent, and its sources continue next run. The same holds at once for a host whose Retry-After asks for more than the five-minute cap on a wait, and for a host that times out on two requests in a row after their retries, so a hanging index cannot spend the run's time budget.
• Overlap. Each pull overlaps the last by overlap_days to catch late-indexed captures.
• Analysis window. Every run analyses the stored records captured in the last analysis_window_days together, so clusters can span runs, plus the run's new records whatever their capture time: an archive that indexes a capture late still gets it analysed once. The digest notes how many new records were captured before the window, and run.json counts them. The overlap before the window is read for the archive burst baseline only. case new defaults to the same window, so a cluster number from the digest names the same cluster there; a record named with --record is taken from the whole store when it is older.
• Changed queries. Each source's state holds a fingerprint of its kind, pattern, match type, filters and endpoint. A source edited under the same name starts over (cursor, seen IDs, resume key and checkpoint reset, health kept), records its store already holds are not reported again, and the digest says so. A state saved without a fingerprint counts as matching.
• Removed sources. The state of a source no longer in the watchlist is set aside under retired in state.json, with a digest note; a source added back under the same name continues from it.
• urlquery. urlquery sources need URLQUERY_APIKEY. Without it they are skipped and the digest says so (configuration).
• GitHub search. ghsearch sources need GITHUB_TOKEN. Without it they are skipped and the digest says so.
• HTTP cache. Responses are kept in cache/http/ and reused for http_cache_ttl_secs (default 3600, 0 turns it off), then revalidated with their ETag. Requests that carry an API key are not cached. Each entry holds its body's SHA-256; a body that does not match is a miss. Cached bodies are raw answers, not redacted records, so every run deletes entries older than the TTL, and at least a day (retention).
• Store lines. A store file is read line by line as bytes. A line that is not UTF-8 or not a record (a write cut short) is moved to <partition>.damaged beside its file the first time a run sees it, with a store_damaged health alert; the other lines are read, and later runs do not alert on it again. Retention deletes a .damaged file once it was last written before the cutoff. Before an append, a last line without its newline is ended, so it never swallows the next record.
• Deadline. Collection stops starting sources after collect_deadline_mins (default 90; 0 turns it off). A source already pulling finishes; the rest are reported as deferred, with what their absence costs, and go first next run. Analysis and the digest still run.
• ASN. Once swarmchase asn fetch has put the IP-to-ASN table in asn/, records that carry an IP but no network get one (configuration).
• Network ranges. Each run refreshes netranges/ (below) before it collects, then describes the analysis's clusters from it.
• Redaction. Collectors redact what they return, and the watch redacts every pulled record again before it reaches the run directory or the store, so a collector that misses a field cannot put a secret in the store. Redaction is idempotent: a record redacted once passes through unchanged.

Network ranges#

netranges/ holds the address ranges AWS, Google Cloud, Oracle Cloud and Cloudflare publish for their networks and the Tor Project's bulk exit list. Each run fetches a file whose held copy is a day old or older, checks that the download parses, and only then replaces the held copy; a failure is a digest note and the held copy stays in use. swarmchase netranges fetch downloads them all now, status shows each file's age and lookup <ip>... prints an address's hosting class and Tor flag. The analysis describes each cluster's client addresses from them (hosting class). The files are never committed.

Source kinds#

Each [[source]] names a kind. Its pattern means what the table says; endpoint and scan apply where noted.

Substrings are |-separated and case-insensitive. Archive kinds date records by capture, scan services by submission, and the rest by when the request reached the service (ts_kind).

Streamed and feed kinds read a bounded amount each run and report their coverage, and their records are leads for review: the record's URL is the event, package, certificate, footprint or tool server, so the URL rules rarely fire on it. The wellknown kind reads only Wayback index rows for agent cards, agent registration files and signature key directories, never the files. The mcp kind keeps a server's name, version, publish time and declared hosts as reference data for linking.

GH Archive and Wikimedia notes#

GitHub's event payloads are trimmed: a pull request carries its number and refs but no title, body, link or author association, and a push carries no commits. A gharchive source therefore finds pasted URLs in issues, comments, reviews, discussions and releases, and run tags in repository names; a pull request's record links to it by number. scan hours at most are read per run, and max_mb is sized to fit them, so the hour cap ends a run.

A ghsearch source fills the gap GitHub's trimmed payloads leave: it searches pull requests created and commits committed since its checkpoint, one search per substring, and checks every result against the substring again, since search matches words. Only matching links and run tags become records, the link as URL and the pull request or commit page as reference; the footers coding agents add to their own work never match.

Each search reads oldest first, from three hours before the checkpoint (for items indexed late and commits dated before their push; seen IDs drop the repeats) to ten minutes ago. The checkpoint moves only as far as every search read: a search cut short by the request budget, a failed request or GitHub's 1000-result cap holds it at the newest item that search reached, and the next run reads on from there. A rate limit GitHub answers with HTTP 403 is waited out like a 429, within the client's retry cap. A search GitHub answered with incomplete_results counts as skipped, and so does every search when a checkpoint older than 48 hours is skipped to the newest six hours.

A clawhub source that finds more matching skills than its limit keeps the oldest and moves its checkpoint only to the newest of those, so the rest are read on the next run; the pull prints a warning when that happens. A listing page after the first that fails or does not parse stops the walk: the skills read so far are kept, the checkpoint stays where it was, and the source shows as stopped, so the next run reads the gap again. Only pages running out at scan count the rest of the span as skipped.

A bluesky source reads Jetstream's posts for a sampled stretch: from its checkpoint, or the newest half hour when the checkpoint is older, for at most scan wall-clock minutes. Links posted outside the stretch are never seen, and the digest's coverage shows minutes read against minutes since the last run. Each link in a post's link facets or link card that matches is a record whose reference is the post's page; a post deleted during the read is dropped.

A wikimedia source reads the mediawiki.page-links-change stream from the millisecond after the last event read, until it reaches the time the run began or runs out of scan wall-clock minutes or max_mb megabytes; the rest waits for the next run. Each added external link is a record whose URL is the link, never fetched, and whose reference is the edit's diff. Bot-flagged accounts and the stream's canary events are dropped. filter takes domain:<wiki>, !domain:<wiki> and ns:<n>. Coverage is in minutes of stream time. Requests carry the client's identifying user agent, as Wikimedia's User-Agent policy asks. Every kind's terms of use are summarised in the watchlist and in its collector.

The record store#

The store holds one directory per source and one file per UTC month of capture time, store/<source>/<YYYY-MM>.ndjson. A record with no capture time goes to store/<source>/undated.ndjson: its age is unknown, so only a read with no lower bound takes it in (case add --record, inbox) and retention keeps it.

• Windowed reads. A run reads the months its analysis window and overlap reach and opens no older file. A record left over from a run cut short is found by ID in its source's files, most recently written first, without reading the rest.
• Damage checks. Each run checks the files it reads and every file written to since the window began, since a damaged line comes from a write cut short.
• Older layout. A data directory with one file per source (store/<source>.ndjson) is moved to this layout at the start of the next watch run, under the data lock, with a one-time note in the digest. Each old file is split into its months, each month file rewritten whole and deduplicated by ID, and the old file deleted last, so a run cut short during the move loses and duplicates nothing and the next run finishes it. Unreadable lines go to undated.ndjson as they are and are set aside from there with an alert; an old <source>.damaged becomes store/<source>/legacy.damaged. Until then every read takes in the old files too.

One writer at a time#

A watch run holds an exclusive lock on the data directory (.lock) from start to finish; case new, case add, case rename, a case opened from review (c), canary register, watch reset, watch prune-cache, backup create and backup restore take it too. A second command that needs it fails at once, naming the holder from .lock.owner, instead of writing beside the first. The lock is advisory and released when the command exits, however it exits. state.json and every other rewritten file are written to a temporary file of their own and renamed into place; the previous state.json is kept as state.json.bak, read with a warning when the current one does not parse.

A review file is saved under a lock of its own and merged: a save rereads the file and writes only what that command changed, so a review --serve page left open never undoes a review tuned run meanwhile.

Interrupted runs#

A run saves state.json after each source, with the IDs of the records it stored under undigested. They are cleared only once the run's digest and run.json are written and handed to the workflow ($GITHUB_OUTPUT). Case hits, the rule and slice history and alert streaks are saved in that same write (the saves after each source keep each source's health as the run found it), and the ledger and autonomy.ndjson just before it, so a run cut short before its digest leaves them as they were and the next run reports its case matches again. A run cut short after collecting (killed, out of time, a crash) leaves them, and the next run loads those records from the store and reports them as new, with a note in the digest and their number as carried_over in run.json.

Steps after collection do not stop a run: a file that cannot be written or a history that cannot be read becomes a step_failed health alert, and the run goes on to its digest. A step that fails run after run alerts as other repeated failures do (health), with its streak in state.json (step_failed). A summary.json or ledger.json that does not parse is read from its .bak copy; when that fails too, the damaged file is kept as .damaged, the history starts afresh and the digest says so.

A step after the digest that fails (the public log, state.json, the run's summary.json, LATEST, retention) ends the run with an error, having lost nothing the digest reported: the workflow already has changed, so the digest is mailed. Should state.json fail to save, the next run reports the records again.

One source, or starting one over#

watch --only <name> pulls one source, with its cursor, health and undigested records as a full run keeps them, and leaves the others as they are; the analysis and digest cover the store as usual. watch reset <name>… makes sources start over on their next run: it clears each one's cursor, seen IDs, resume key and checkpoint under the data lock, and keeps its health history. The next run drops the records the source's store already holds, as for a changed query, so they are neither stored twice nor reported again.

watch prune-cache [watchlist.toml] deletes cached responses older than the TTL (at least a day), or older than --max-age-hours, as each run does.

Checking a watchlist#

swarmchase watch --check [watchlist.toml] reads a watchlist strictly and exits: a key no setting knows (a misspelt limit that a run would ignore, leaving the default), a value of the wrong type, a duplicate source name, an [autonomy] level that cannot be on, and every source built as a run builds it (its kind, endpoint and other settings). It reads no data directory and sends no request. CI and the pre-commit hook run it on the committed watchlist (scripts).

The digest#

Each run writes runs/<stamp>/digest.md. It opens with the run's priority and source health, then covers new clusters, tier changes and endpoints hit for the first time. When something changed, the scheduled workflow emails it to the admin, with the priority in the subject ([swarmchase] [HIGH] watch digest <stamp>); nothing else leaves the runner.

Needs attention#

The first block gives the run a priority, high, medium or low, and lists the facts that set it, highest first. Only deterministic facts count:

A lead is a cluster above noise that is new since the last run or rose a tier, or a new significant record that joined no cluster. New and risen are judged against the cluster's ledger thread: a cluster whose key changed because its earliest record left the window is compared with its own previous run, not reported as new. A case clock has run when reports were sent and the indicator notice's lead time has passed with the notice not yet added, or when the notice is added and an operator may now be named. Each due clock is also a Case status line with the command that starts the step; the watch takes no step itself.

The leads of one campaign are one reason of kind campaign, at the level of its strongest lead and in its place: the campaign, its site, its new leads (clusters and records outside any cluster) and its strongest lead. One operator's many sessions are one item, not one per session.

A model's suggestion never sets the priority: the same facts give the same priority whatever a model said. A new significant record outside any cluster at L2 or higher makes the run count as changed, like a new cluster.

Attention queue#

The next block counts the attention queue by band and lists up to eight leads at urgent or look: the cluster, grade, records, trajectory, the facts that set its band, any caps, targets, momentum and covering case. Under it, each line an autonomy level wrote this run, marked shadow (would have) or on (did).

Health#

The health block comes first, because a source that did not run hides whatever it would have found. Its alerts are:

• a source that failed this run, with a short reason such as HTTP 504 after 3 attempts, and how many runs in a row it has failed. One failure is enough, so a failure reaches the admin within one run;
• a host that stayed unavailable through its retries, so the run stopped sending to it;
• a source whose canary failed: it answered, but nothing came back in the shape the parser expects;
• a source skipped for the first time (no API key);
• a source whose pull falls below a fifth of its median over recent runs, which usually means an API change, an expired key or a spent quota. A source's first pull and a pull spread over several runs are catching up, so they do not count towards the median;
• a rule whose hit rate on one source's new records jumps to three times its recent rate on that source. Each source is compared with itself, because a relay's own rule fires on nearly every record from it. When the rule's slice is also rising as a slice onset, the alert reads "drift or onset", since a swarm starting below the clustering threshold looks the same; when most of the source's slices rose together, it reads as a source-side change;
• a store file with lines that cannot be read, with their count. They are moved aside; the other records are read;
• a step after collection that failed and was skipped, or a summary.json or ledger.json started afresh because it could not be read (interrupted runs).

Under Capability lost this run it lists each skipped, failed or stopped source with what its absence costs, and the part of a stream it skipped. When no submit-time source ran (urlquery is the only one scheduled), it says so first: without submission times, timing links, pace, agent grades above L2 and interaction edges are unavailable. Quiet sources notes a source that has pulled nothing for several runs, and whether its canary passed.

Any alert gets the digest emailed; lost capability and quiet sources alone do not. A failure that repeats (a source, a host, a canary, a step after collection) alerts on the 1st, 2nd, 4th, 8th and later powers of two of its streak; on the runs between, the sources table still shows the error and a note gives the streak, without mailing a digest for it.

A run in which no source could run, or every source that ran failed, ends with an error (exit 1, the workflow's failure mail) on the same runs of its streak (failed_runs in state.json), and with a digest note on the others. Such a run still writes LATEST; only a run in which a source worked writes LATEST_OK, which the staleness check reads. A failed source's reason never includes the request URL; the full error is in the run's log and in state.json.

Digests not mailed#

The workflow's mail step writes runs/<stamp>/mailed once a digest is sent. A changed run newer than the newest mailed one, without the marker, had a digest that never reached the admin (no mail that run, a refused send, a cancelled job). The next digest lists those runs under Digests not mailed, before the health block, with each one's priority reasons and leads, takes the highest of their priorities as a reason of kind unmailed, and counts as changed. A data directory with no marker at all is not mailed from, and nothing is listed. Each run keeps the newest marked stamp in state.json (last_mailed), so retention deleting the run that held the marker does not stop the list. The dashboard lists the same runs.

The newest twelve are listed; older ones are counted in the section's heading and a line of their own.

Slice onsets#

Some swarms never cluster above noise: one agent per tag, a few records a run, each below tier. What they change is a count. Each run counts its new records per slice of each source: per rule they hit and, for flagged records, per specific scaffold template and per target site family (a host's site, methodology). It keeps the last 16 runs per source in state.json, each with the source's records that run (its exposure) and a few example record IDs per slice.

A slice is rising when a Poisson CUSUM over its newest runs (up to six), against its rate per record over the older runs, reaches the threshold (methodology). A source that returned more rows at the same rate is not rising. Fewer than three older runs, or fewer than 100 records under them, is no alarm.

Slice onsets lists each rising slice: which source and slice, the records in it since the run it started against the source's records over the same runs, the baseline counts and rate, the start run, and example record IDs. Hosts and IDs are defanged. When the source's volume also changed fourfold, the line says so. When at least three of a source's established slices (ones its baseline saw), and most of them, rise together, the section lists one source-side change (pagination, a new crawl, a recovery) instead.

An onset is a lead in the mailed digest only, never a public artifact. It adds a medium priority reason of kind onset (a source-side change one of kind source_change, at low) and makes the run count as changed. A slice still rising is mailed again on the 2nd, 4th, 8th... run of its streak, as health alerts are; the section shows the runs between, marked as repeats. The history restarts when the rules change, and runs older than retain_days leave it with their example IDs.

Sources and coverage#

The sources table shows each source's records, requests, coverage and status. Coverage is what a stream or feed source read against what there was to read since its checkpoint, such as 6 of 6 hours, canary ok or 1024 of 400000 entries, 398976 skipped for a CT log that grows faster than a polite reader.

The digest ends with a Run time line: the run's time up to the digest, each phase (collect, platform, analysis, triage) and the three slowest sources. run.json has the same in milliseconds. An HTTP cache line follows: responses served from the cache this run, entries kept and their size, and entries deleted.

Headline counts#

The window's headline counts records with an indicator beyond the relay or carrier they came through: a hit whose kind is not a route kind (relay, cache buster, self-id, header, outcome) and is not a carrier rule naming the record's own host. Every watch source is a relay's or carrier's own host, so a plain "has a hit" count would equal the records pulled. Clusters are counted above noise; noise-tier clusters are listed in their own section and left out of the count.

Significant records outside any cluster lists significant records that joined no cluster, new ones first, with their rule ids, tier and defanged target host, so a lone strong record is not reduced to a host name.

Possible reruns and archive bursts#

• Possible reruns. A new run whose target hosts, and their order, repeat an earlier run in a different cluster on another day. Hosts common to many runs count for little. The thresholds are in methodology.
• Archive bursts. Many Wayback or Arquivo captures of one host within minutes, on a host the store otherwise holds few captures of, measured over the span the run reads, with how many carry an indicator beyond the relay. The archive_burst_* settings in the rules file set the thresholds.

Both are leads to review; neither merges clusters.

Campaigns, candidate fleets, escalations and common features#

Over the whole window, up to ten lines each:

• Campaigns. One site's senders holding one habit set beyond chance: the site, the core, its members out of the site's senders, the holders anywhere, the chance, and the member clusters and records outside any cluster.
• Candidate fleets. Clusters tied by soft links, with the kinds and the weakest pair's strength. Merging one is a case decision (fleets). The same fleets by their keys across runs are under fleet threads.
• Escalations. Requests for one page whose route climbed (direct, relay, program, remote browser), with the clusters they touch.
• Common features. Linking values shared too widely to link on, so they were demoted: records, hosts, span and the limit they passed.
• Noise-tier clusters. Clusters at noise tier, listed but not counted in the headline.

Clusters are named by number and stable key throughout (C3 k-3f9a2c1b), and hosts and values are defanged. These sections alone never make a run count as changed.

Fleet threads#

Each candidate fleet of the run by its key (FK-..., its number, new or the runs seen since its first), its member threads by cluster and key, its weakest edge ratio and density. Then the persistent pairs no current candidate fleet holds together: the two threads, the accumulated log ratio, the windows that counted on their own, the runs that linked them and the latest link kinds. Up to ten lines each; keys and kinds only, no host or value. Proposals only, like candidate fleets; the section never makes a run count as changed.

Model triage#

Off by default. The watch asks a model for a label on each new lead's strongest record, through one of two backends:

• Local. watch --triage (or --triage=local, or SWARMCHASE_WATCH_TRIAGE set to local, 1 or true) with SWARMCHASE_LLM_URL set asks the model at that URL, one record per ask. It uses the client of label suggestions, with the same rules: a loopback, private or link-local endpoint only, defanged text, no IP addresses, no tools, no proxy, no redirect. It may see any lead.
• Claude. watch --triage=claude (or SWARMCHASE_WATCH_TRIAGE=claude) asks Claude through Claude Code, logged in as the user running the watch (runner). Claude is a hosted model, so it is sent only leads from the hosted sources, as the lines review --export-hosted writes, with their grade, agent_allowed, signals and rules; the rest are counted as withheld. All of a run's leads go in one call, or one per 25 records.

The rest is common to both:

• Limits. Leads are asked strongest first, up to --triage-max per run (default 20, SWARMCHASE_TRIAGE_MAX) and within --triage-secs (default 600, SWARMCHASE_TRIAGE_SECS). A local ask takes at most 60 s, and after three failed asks in a row the rest are left for the next run. A Claude call takes at most the time left; after two failed calls in a row, or a usage or rate limit, it asks no more that run.
• Failures. A missing or refused endpoint, a missing claude, a timeout, a failed call or a reply that is not the asked-for JSON is no suggestion. The run carries on and the digest says how many were asked, answered, left and withheld, and why it stopped.
• Saved. Answers are appended to labels/suggestions-local.ndjson or labels/suggestions-claude-watch.ndjson, one {"id", "label", "reason", "model", "run"} per line, the format review reads; Claude's lines also carry grade and misfires. A record already in the file is not asked again; its earlier suggestion is shown. Review reads Claude's file as a hosted model's, so a line for a record from any other source is dropped.
• Shown. Model suggestions names the backend and lists each lead's suggestion with its tier and agent grade. An agent suggestion on a lead below L2 is marked as against the labelling rule, under which unsure is the most that grade allows.

A suggestion is never a label: review saves it beside the admin's label as suggested, and eval reads only the admin's.

Canary sightings#

When the registry holds tokens (page references), each run searches its new records for them: the URL and every layer unwrapped from it, the final URL, the user agent and the header values. Each copy is a line under Canary sightings with the token, the field, the record id and source, the file it was issued on and its month, and a medium priority reason of kind canary. A token-shaped value the registry does not hold is listed as unverified. A record dated before its token's month is marked: the seed has leaked, the value was made up, or the record's time is wrong. Sightings never link records and never become leads in the attention queue. With an empty registry the check does nothing.

Case status#

The watch matches new records against every case's indicators: its clusters' linking features and its endpoints, under the case's pinned rules. The Case status section then proposes:

• marking an active notice contained after contained_after_days (in watchlist.toml, default 14) without a matching record;
• reopening a contained or historical notice, or a closed case, that has new matches;
• re-checking cases whose records would change tier under the current rules.

The watch changes nothing itself. Apply an approved proposal with swarmchase notice update <id> --kind status --status <s>, which also updates the case.

Platform swarms#

A separate pass in each run looks for accounts acting alike on GitHub, GitLab.com, Codeberg, package registries (npm, PyPI, NuGet, Packagist, RubyGems, crates.io, Go modules), the Hugging Face Hub, Wikimedia wikis, Hacker News, Bluesky and CVE records: near-identical or templated text, templated names, or text-less stars, forks and pull requests on one target, from several accounts in a burst beyond chance. The method and its thresholds are in methodology. A failure in this pass is reported in its section and never stops the record analysis.

Sources. The watchlist's [platform] table holds carry_hours (default 72), label_days (default 30), history_days (default 30), provenance_lookups (default 40), optional [platform.options] (the detector's tunables, such as min_accounts, burst_secs and max_chance) and [[platform.source]] entries. Each has a name, a kind and the record source fields it uses:

pattern is required by the format; only npm and pypi read it. Sources that read from a checkpoint (all but npm, pypi, huggingface and rubygems; openvsx keeps one but reads one page) keep it in platform/state.json; a backlog over 24 hours (a week for osv and wikimedia-ai-cleanup, half an hour of stream for bluesky) is skipped to the newest stretch, and the section shows how much was read and skipped. A corrupt GH Archive hour is skipped and counted.

Labels. An osv or wikimedia-ai-cleanup source gives labels, not events: the packages OSV's malicious-package records name, or the wiki pages editors tagged as suspected AI-generated text. They are kept in platform/labels.ndjson for label_days (default 30). A candidate shown in the digest whose members published a labelled package or edited a labelled page gets a Labelled line with the records behind it, and runs/<stamp>/platform-labels.json lists every candidate's matches.

Build provenance. For the npm and PyPI candidates new or grown this run, the pass looks up the build provenance of member releases: npm's attestations and PyPI's integrity API, by name and version, at most provenance_lookups a run (0 turns them off), one member per candidate in turn, members on an OSV-labelled package first. Only the source repository, workflow, ref, commit, runner and deployment environment are kept; no package file is downloaded. Answers, including "none", go to platform/provenance.ndjson for label_days, so a release is looked up once.

Each npm or PyPI candidate with answers carries them as provenance in platform-swarms.json, and the digest gives it a Provenance line: the repositories and workflows its packages name, whether several packages share one, or that none carries provenance (weak context). It changes neither the chance nor the strength (methodology).

Declared coding agents. GitHub events from the coding-agent Apps that declared-agents.toml lists (github_app) are tallied by product on the source's line, not grouped. An undeclared App whose login carries one of those products' names is listed below it as a spoofing lead, by login: App logins name software, not people. A provider's name alone claims no product (openai-sdks[bot] is no lead), and an App acting on a repository of the provider's own GitHub organisations is the provider's own (identity).

What is stored. Nothing from the full stream. After grouping, only events worth another look are carried in platform/carry.ndjson for carry_hours: members of a group with two or more accounts and events carrying residue. Events hold a keyed hash for the account, never its login, and slotted text sketches, never the text. The run's candidates, suppressed groups and counts go to runs/<stamp>/platform-swarms.json.

The burst baseline. platform/baseline.json keeps what earlier runs saw: each platform's stretches read (a pull's first event to its last) and the model burst chance of every group weighed, for carry_hours, and the name and text templates of members in a burst, with the window they burst in, for history_days. Bursts are weighed over the time read; a platform read for under an hour has its bursts not weighed; a burst chance is never below the share of earlier groups on the platform that burst as tightly; and a group whose families burst in an earlier window is suppressed as a recurring family unless its accounts are new or its text carries residue (methodology). Deleting the file starts the baseline afresh.

In the digest. Platform swarm hints lists candidates that are new this run or have grown since they were last listed, each with its reason line (burst, new accounts, residue and their chances), what joined it, the slotted template, its targets, any coding-harness self-identification (weak context), its packages' build provenance, what is left to rule out, and up to five defanged links. Every moderate candidate is shown, and the first 10 weak ones; the rest are summed by platform, and every candidate stays in platform-swarms.json. A new or grown candidate counts as a change. Suppressed groups are summed by benign kind. Candidates are not leads for triage: the hosted model may see only archive and urlquery records (policies).

On demand. swarmchase platform-swarms <events.ndjson> runs the detector over carried events or fixtures, and --gharchive <hour.json.gz> over GH Archive hour files read locally; --json prints the full report, --options takes a TOML of tunables and --baseline a watch's platform/baseline.json.

The run summary#

Each run also writes runs/<stamp>/run.json, the run in a small JSON for scripts, local dashboards and agents. LATEST names the newest run's stamp. The file stays in the data directory with the digest. Hosts in it are defanged, and it carries no record URL, IP address or model prompt. schema is swarmchase-watch-run/1; fields are only added within a version.

A reason has level, kind (lead, case_clock, unmailed, canary, onset, source_change, or an alert kind) and the digest's text. An alert has kind (source_failed, source_skipped, source_collapsed, canary_failed, host_down, rule_drift, store_damaged, step_failed or other), the source or host it concerns, and text. A source's status is complete, partial, stopped, error, skipped or deferred. A due clock's clock is indicators or naming, and since is Unix seconds.

Lead fields#

The public log#

watch --public-log <path> also writes a summary safe to show in a public CI log, and the scheduled workflow prints only that. It is built from an allowlist, not filtered from the run's output: one line per source with its name and kind (both from the watchlist), status word, pulled, new and requests counts and duration_ms, then a total line with the same counts, sources per status and the run's duration. Nothing in it is read from a record, an analysis, a lead, an error message or the priority. A name is printed with anything but letters, digits, -, _ and . replaced, so it cannot carry a line break or a workflow command.

The ledger#

Each run analyses its window afresh: cluster numbers change, and a cluster's key changes when its earliest member leaves the window. ledger.json follows each cluster above noise across runs as a thread, so its history outlives the window. Code: ledger.rs.

• Following. A cluster continues the thread whose key it has, else the thread it shares the most member records with. Each thread keeps the latest 200 member ids for this.
• Points. One per run the thread was in the analysis: tier, grade, records, records that run collected (added), targets, interaction edges and attention band.
• Events. A new key, a merge (another thread's records joined it), a split, a rise in tier or grade, and a return.
• Returns. Linking features join records only within days, so an operator using the same inbox or tag weeks later starts a new cluster. A new thread carrying an earlier thread's linking feature names it in recurs, and after a quiet week or more the run records a return. Event text names the feature, never its value. Nothing is merged.
• Trajectory. On the latest run a thread is new, back after a quiet spell, growing (at least twice its records of a day before, and three more), active (records collected in the last day), quiet, or dormant (not in the analysis).
• Fleets. Each candidate fleet's member threads are matched to the earlier fleets' (Jaccard at least 0.5, one to one, largest overlap first). A match keeps its key; a new fleet takes FK- and 8 hex digits of a hash of its earliest member thread, with a suffix if taken. A key is never given to another fleet. Each keeps its members, first and latest run, runs seen, its largest size, and on its latest run its number, link kinds, weakest edge ratio and density (SC-FM 5.18). Code: fleet_track.rs.
• Pair evidence. Each pair of threads a run links keeps the log of its combined soft-link ratio across runs. Overlapping windows count once: a run adds only when every record of the pair's two clusters comes after the records of the run counted last, and otherwise replaces that run's count when stronger. The total halves every 14 days. A pair is persistent at a total of ln(1000) (about 6.9), the conservative upper threshold of a sequential test at error rate 0.001.

Fleets and pair tracks are pruned with the threads, under the watch's retention. The ledger is descriptive. It changes no score, tier, grade or link.

The attention queue#

Each run ranks every cluster above noise into runs/<stamp>/attention.json. The rank is layered, never one weighted sum. Code: attention.rs.

1. Band, set like the agent grade by the strongest combination of facts, then held down by caps.
2. Within a band, the grade (L1 ranks with L0), then the worst target class, then clusters no case covers before those a case does.
3. Last, momentum, which orders leads the facts above leave tied and never moves one across a band.

Target classes come from the scope pass; benchmark targets count as none.

Provenance monoculture is left out because nearly every public record comes from a service anyone can submit to; it would hold every lead.

Momentum runs from 0 to 100. Each part is bounded, and the queue shows each with its points and reading:

Nothing in the queue reads a model's suggestion.

Autonomy#

Autonomy levels are things the watch could do on its own about a lead in the queue. Each is internal and reversible. None sends, publishes, names or opens a case. Code: autonomy.rs.

Every trigger needs grade L2 or higher, since anyone can write a self-description. interrupt fires at most once a day per thread and interrupts_per_day times in all (default 3; the rest are logged as held back). watch_harder fires once a week per thread, draft_case once per thread.

The watchlist sets the modes:

[autonomy]
interrupt = "shadow"      # off | shadow | on
watch_harder = "shadow"   # off | shadow
draft_case = "shadow"     # off | shadow
interrupts_per_day = 3

Every level is in shadow by default: it only writes what it would have done to autonomy.ndjson. interrupt = "on" adds a high reason of kind attention to the run's priority and makes the digest count as changed; nothing else acts.

Reading a level's record#

The dashboard reads each line against what the admin decided later:

• Agreed: a case covers the lead, or, for interrupt and watch_harder, a record of it is labelled agent.
• Disagreed: its records were labelled with no agent label and no case, or, for draft_case, no case followed in two weeks.
• Pending: neither yet.

A lead's thread counts with every thread it was merged into. For draft_case, never proposed counts threads a case covers that the level did not propose. Switch a level on when its record shows it would have been right; the log is the evidence for that decision.

Retention#

At the end of each run, run directories and stored records older than retain_days (default 180, by capture time) are deleted: a month file whose month ended before the cutoff as a whole file, unread, and the older records of the month the cutoff falls in line by line. Undated records are kept, and a source directory left empty is removed. Ledger points, autonomy lines and summary.json endpoints last seen before the cutoff go too. Only the newest keep_analysis_runs runs (default 8; 0 keeps all) keep their full analysis.json and case-notes.md, which grow with the window; older runs keep their digest, run.json, summary.json, attention queue and new records. The dashboard reads the newest run's analysis, and a case analyses its own records, so neither needs the older ones.

The platform pass keeps its carried events for carry_hours only. Cached HTTP responses older than http_cache_ttl_secs, and at least a day, are deleted on every run, before the digest; watch prune-cache does the same on demand. Copy anything a case needs into the case first (case new or case add). The policy is in policies.

The dashboard#

swarmchase dashboard shows the state of monitoring on http://127.0.0.1:8767/ (--port), one of the local pages. It reads the data directory (-d, as for review) and the watchlist (--watchlist, default watchlist.toml, for each source's kind) and writes nothing. It reloads when a new run lands, and r reloads by hand.

swarmchase dashboard -d "$SWARMCHASE_DATA"
swarmchase dashboard -d "$SWARMCHASE_DATA" --json --since 20261011T060000Z

When the latest run wrote a run summary, the page opens with the digest's priority and its reasons, reads the health alerts from it, and lists the local model's suggestions on the run's leads, flagging an agent suggestion below L2 as against the labelling rule.

It answers these questions:

• What needs you first. The attention queue, filterable by band. Each lead shows its band, grade, trajectory, the facts that placed it, caps, targets, a sparkline of its records per run and a momentum bar; Why this place lists the momentum parts and the command that opens a case.
• What is new. Clusters above noise that are new or rose a tier, with their agent grade, and significant records outside any cluster, collected since a baseline run. The baseline is the run before the latest, as in the digest, or the run the admin marked seen (s on the page, remembered in that browser only; --since for --json). When that run was deleted by retention, the nearest kept run before it is used.
• Whether monitoring is healthy. Each source's status from state.json (failing, skipped, quiet, catching up, ok), its recent pulls, what its absence loses, the latest digest's health alerts, and rule drift recomputed on the latest run.
• What waits on the admin. The review queue as review --serve builds it from the latest run's analysis, with label progress and the agent labels below L2 counted apart, and each open case's clocks and what case status says it still needs.
• How leads move. The activity timeline draws each thread from the ledger as a row over the window: a bar per run, its height the records, its colour the band, solid when that run collected records for it, with dots for events.
• What ties leads together. The link graph draws the latest analysis's clusters above noise that something ties: soft links as grey lines, thicker for a stronger pair (combined as candidate fleets combine them), possible interaction (S1 to S6) as red dashed lines, interaction inside one cluster as a ring, and candidate fleets as shaded areas. Groups of tied clusters are laid out apart, largest first, and the same overview always draws the same picture. It can show only interaction or only fleets; selecting a cluster or a line lists each tie with its strength and defanged detail, and the table under it lists every tie. Up to 60 clusters are drawn, the queue's top ones first. A tie is a lead to check, never a merge.
• How autonomy would have done. Each level's mode and record (agreed, disagreed, pending, held back, never proposed) and its latest lines with their outcome (reading a level's record).
• What never reached the admin. Changed runs whose digest was not mailed, as the next digest folds them in (digests not mailed).
• What to run next. Each item comes with its command, with the data directory filled in: review --serve, case … new --cluster <key>, a case's next step, or scan on a drifting source's records.

d shows the latest digest; 1 to 8 jump to the sections; ? lists the keys.

The overview document#

The page reads one JSON document, GET /api/overview (/api/overview/since/<stamp> for another baseline). --json prints the same document, for a script or an agent. Hosts in it are defanged and no record URL is included. schema names the version; a field changes meaning or goes only with a new one.

Reviewing leads#

review steps through the watch store's recent records, strongest cluster first, and asks for a label.

Text after the key is saved as a note (b ordinary article read). Answers are saved one at a time, so quitting loses nothing. A label line that does not read (a write cut short, a hand edit) is skipped with a warning, and the next save moves it to <file>.damaged beside the review file.

Each record shows its agent grade. Labelling agent below L2 prints the grade, its signals and caps and the labelling rule, then asks for a reason, saved as the note: Enter saves the label with none, and u labels unsure instead. A note typed after the key is taken as the reason.

• --unflagged N adds a stable sample of unflagged records, which grows the benign baseline and catches misses.
• review sample writes a stratified, seeded sample as a queue file, and review <queue> --all labels every record in it (a reviewed benign baseline).
• --stats counts reviews and disagreements, and Cohen's κ between labellers; --list prints the queue.
• --labeller <name> saves to labels/reviewed-<name>.ndjson instead, for a second, independent reviewer (two labellers).
• --suggest shows a local model's label suggestion before each prompt (label suggestions).
• --serve labels the same queue in a browser on this machine (the review page); --export-hosted <file> writes it for a hosted model's suggestions (hosted suggestions).

Each answer goes to labels/reviewed.ndjson with the record, so swarmchase eval <data>/labels/reviewed.ndjson measures the rules against everything reviewed (evaluation). The file stays in the data directory and is never committed.

Label lines#

Each line is the stored record with these fields beside it:

The grade on a label#

grade is the grade that decides the record's labels when it is labelled: its cluster's, else its own. It holds code and label (absent for an ungraded record), basis (cluster, record or none), cluster, signals and counter as <signal> (<n> records), caps, and agent_allowed. An agent label below L2 also carries "rule": "below_l2": an exception the admin chose, with the reason in note (evaluating leads).

A line without grade comes from before grades were stored. Its grade is unknown, so each reader grades it from the analysis it has and says so. The review page and the dashboard use their own analysis. review --stats and review agreement analyse the review file's own records, so a cluster with members outside the file may grade differently. eval uses the analysis of its inputs and does not judge a line whose record is not among them.

review --stats lists the agent labels below L2 with their grades and reasons; review --stats --json prints the counts, the baseline and the agreement between labellers as one JSON document, with record IDs and counts only. review agreement lists them too, and reads the stored grade before the grade now when a suggestion line records none.

Label suggestions#

review --suggest asks a language model for a label before each prompt and shows it as suggestion: agent, <reason>. The admin still answers; the suggestion is saved beside the label as suggested and is never the label, so eval reads only the admin's.

It is allowed only with a model running locally:

• Local only. SWARMCHASE_LLM_URL is an OpenAI-compatible server (/v1/chat/completions; the base URL is enough) on localhost, a loopback address, a private address (RFC 1918, or RFC 4193 for IPv6) or a link-local one. Any other host is refused before anything is sent, so records never reach a third-party API. The request goes through no proxy and follows no redirect. SWARMCHASE_LLM_MODEL names the model (configuration).
• What it sees. The defanged URL, target host and rule hits, the tier and score, and the cluster's size, tier, top rules and target hosts, with a fixed prompt asking for {"label": "agent" | "benign" | "unsure", "reason": "…"}. No IP addresses.
• No fetching. The model gets text and no tools; the prompt says the URLs are data, never to be visited.
• Failures. A timeout (60 s), an error or a reply that is not that JSON means no suggestion for the record; the review carries on.

Hosted suggestions#

A hosted model may suggest labels for public records only (policies):

swarmchase review --export-hosted data/labels/hosted.ndjson          # the flagged queue
swarmchase review <queue> --all --export-hosted data/labels/hosted-baseline.ndjson

• What it writes. One {"id", "group", "text", "grade", "grade_basis", "agent_allowed", "signals", "counter", "caps", "rules"} per record: the text the local model reads, the record's review group, the agent grade it is labelled against and the rules that fired. --batch-size N splits the export into numbered files and keeps each group in one. Records from wayback, arquivo, memento, commoncrawl and urlquery only; operator logs and the rest are counted as withheld. The list is in code, so a new source is withheld until it is added on purpose.
• Where. The file must be inside the data directory, and stays there.
• Answers. The model's suggestions go in labels/suggestions-<name>.ndjson, one {"id", "label", "reason", "model", "grade", "misfires"} per line (grade and misfires are optional). review suggest-claude <export>... asks Claude through Claude Code, as model triage does, and appends its answers to labels/suggestions-claude-<export name>.ndjson; records already in that file are skipped, so a run cut short by a usage limit resumes. review check-suggestions validates a file against its export. review --serve reads every such file; a line for a record from any other source is dropped, except in suggestions-local.ndjson, which a local model writes (model triage).
• The loop. review agreement compares the admin's labels with each model's, and review misfires lists rules that keep firing on records labelled benign. How to run it all: AI assistance.
• No fetching. Whoever runs the model gives it the file and no web access.

The review page#

review --serve serves the queue on http://127.0.0.1:8765/ (--port) instead of prompting, with the same inputs and options as the terminal session. It is built on the shared local pages.

• Layout. The queue of groups is on the left, with tabs for what is left to do, labels that differ from a suggestion, what is done, and everything, and a search by rule, host or id. The selected group is on the right: its agent grade and signals, its suggestion, the rules that fired, and each record with its decoded layers.
• Groups. Records are grouped by cluster, or by shape (the rules that fired and the start of the URL) when unclustered, so a run of near-identical records is labelled at once. Each record can still be labelled alone.
• Flow. Labelling a group moves to the next one. z undoes the last action, label and note included.
• Triage. The queue sorts in queue order, needs attention first (rule conflicts, then split suggestions, then mixed or differing labels), largest groups first, or by grade. Filters narrow it by grade (L2 or higher, below L2, or one rung), tier, source, suggestion and check. The browser remembers both.
• Labelling rule. agent needs an agent grade of L2 or higher. The server marks each record and group with whether the rule allows it, a suggestion of agent below L2, and a saved agent label below L2. The page shows these in the queue and on the group, and a group labelled so can be relabelled unsure in one step.
• Exceptions. Labelling agent below L2 opens a dialog with the grade, its signals and caps and the rule, and a reason field saved as the note; it offers unsure instead. The server saves every label with its grade (the grade on a label). An agent label below L2 shows as agent < L2 in the queue and the done tab, with an exception field on each record giving the grade, signals, caps and reason, and in earlier decisions. The top bar counts them, and the session panel lists them with the review file's count and the precision estimate without them. An older line without a stored grade shows the grade now and says so.
• Suggestions. accept suggestion labels the selected group's unlabelled records as suggested, never across groups, and leaves any suggestion of agent below L2 for the admin. The suggestion is saved beside the label as suggested.
• Bulk. A / B / U (with Shift) label only the group's unlabelled records. Every bulk label is one undoable step, and an undo puts back each label with the grade it was saved with.
• Session. p opens the session panel: labels this session and in the review file, agreement between the labels and the suggestions as a table, labels that break the rule, and the agent share of decided flagged records with its 95% Wilson interval, the labels still needed for ±10 points, and the swarmchase eval command to run next.
• Changing labels. Reviewed records stay on the page, so a label can be changed or cleared. Labels go to the same file as the terminal session (--labeller included).
• Rule context. Each rule in the group shows its description, kind, weight and tactic from the rules the analysis ran with, how many records it fired on in the analysis and how many of those were flagged. With a pre-era control summary, it also shows how often the rule fired and flagged before agents existed (negative control). --control <file> names the summary; the default is <data>/controls/pre-era/summary.json, then data/controls/pre-era/summary.json.
• Timeline. When the group's records happened, binned, with the median step and the largest gaps, and whether the times are requests, submissions or archive captures (FM-6.1).
• Earlier decisions. Reviewed records outside the group sharing its rule set, cluster or target host, with their labels and notes.
• Related records. Records anywhere in the analysis sharing a target host or a linking feature with the group, with tier and label, and a jump to their group when it is in the queue.
• Programs and copying. A decoded layer that reads as a program can be laid out as text with its outbound and page-reading calls marked. It is never run or rendered. Buttons copy a record's id, and the group's ids or defanged references, one per line.
• Loading. Context loads when a group is selected, from the analysis, the review file and the control summary only. Nothing is fetched.
• Keys. j / k move, a / b / u label the group, s accepts its suggestion, x clears, z undoes, n writes a note, e shows every record, / searches, f filters, o changes the sort, p opens the session panel, 1 to 4 switch tabs, ? lists them.
• Access. Focus returns to the same control after each label, a dialog keeps focus inside it and gives it back on close, moving between groups is announced to screen readers, and rule flags carry words and an icon as well as colour.
• Guide. g opens the evaluator guide, with worked examples and a practice round that saves nothing, and h turns on blind review, which hides suggestions until a group is labelled. The same method in prose is evaluating leads.

Local pages#

The admin's pages run on this machine from the CLI, never on the public site. They share one server and one look (localui):

• Network. The server listens on 127.0.0.1 only, refuses any other Host (no DNS rebinding), and accepts a change only with the token in the page it served and a same-origin Origin.
• Content. Records are shown defanged, as text nodes, never as links, under a content security policy that loads nothing but the page's own files.
• Look. Shared tokens and components (verdict buttons, chips, cards, keyboard help, toasts) follow the system's light or dark setting.

Source: docs/reference/watch.md