SC-Rules v1: indicator rules format

Specs index · Schema: schemas/sc-rules-v1.json · Conformance vectors: conformance/rules-v1.json

This page specifies the format of a swarmchasing rules file, such as rules/default.toml, and the semantics an engine gives it. A group that writes its own rules, or its own engine, can exchange rules with this project and check its engine against the same vectors. The method the rules serve is the Field Manual (FM-4.1 to FM-4.8).

Must, must not, should and may in bold carry their RFC 2119 meanings.

Status#

An engine that does not implement groups or the context cap must reject a file that sets group, so a file never scores differently on two engines without saying so.

File#

A rules file is TOML. Its JSON value (the TOML read as JSON) must validate against schemas/sc-rules-v1.json. The schema checks structure and value domains. It cannot check that patterns compile or that rule ids are unique; an engine must reject a file with a duplicate rule id or a pattern that does not compile.

A missing settings table takes the defaults the toolkit's bundled file carries, so a case pinned to an older file renders the same. The meaning of each setting is in methodology; this spec fixes only its name and type.

[soft_links.joint] holds caps on two soft-link families' combined likelihood ratio, each a number keyed by the two family names joined by _ in the order harness, timeline, targets, identifiers (timeline_targets). An engine must reject any other key, and must combine a pair's families so that the result is at most each capped pair's cap times the other families' ratios, never below the stronger capped family alone, and the same whatever order the families come in (FM-5.17).

The version is an opaque string. This project uses a date. Two files with the same version must be byte-identical in their rules; a change to any rule, threshold or setting takes a new version.

Rule fields#

A rule must carry a tactic, or artifact = true, or both. T8 and T9 are cluster-level judgments and must not be a rule's tactic (FM-4.3). A check must sit on a rule whose target is the one the check reads. The checks in v1 are repeated_token (a param_value rule: one word of three characters or more with a letter, or a phrase of two or three words, repeated five or more times in a row) and ua_platform_mismatch (a header rule, matching Sec-CH-UA-Platform: the hint names a different desktop system than the request's user agent, with mobile systems never compared).

A rule carries no framework ids of its own: its MITRE ATLAS techniques and OWASP Top 10 for Agentic Applications risks are derived from its tactic through the FM-12.4 crosswalk, so the verified ids live in one table rather than on every rule.

Kinds. relay, carrier, exploit, agent_marker, cache_buster, self_id, exfil, header, outcome, anti_bot, infra. Five of them are context kinds, which ordinary clients produce too: relay, cache_buster, self_id, header, outcome. Context kinds never join records by timing (FM-5.3) and never corroborate a cluster tier (FM-14.9). The default tactic of each kind is FM-4.4.

Layers#

Before rules run, an engine unwraps each record's URL into layers: the submitted URL, the inner URL of each relay or translate proxy, URL-valued parameters, data: bodies and base64 values that decode to text or a URL. The toolkit's rules for unwrapping, and its limits, are in methodology. An engine should unwrap at least relay paths, translate-proxy hosts, URL-valued parameters, data: URLs and base64 path or parameter values, because the default rules assume it; the conformance vectors exercise each.

The target host is the host of the innermost layer whose host is not itself matched by a relay, carrier or exfil rule.

Targets#

Regular expressions are case-insensitive and use the syntax of the Rust regex crate: no look-around and no back-references, inline flags such as (?-i:…), and \x{…} escapes. An engine in another language must use a compatible syntax or translate the patterns, and must pass the conformance vectors.

A pattern with a capture group named m matches as that group: the matched text (what a link = true rule links on) is the group, and the rest of the pattern is context it must see but does not keep. With no look-behind, this is how a pattern says what may not come before a match: (^|[^a-z])(?P<m>agenttest\d*) matches uq_agenttest42 as agenttest42.

Scoring#

For one record:

1. Each rule counts at most once, whichever layer or target it matched.
2. The score is the sum of the weights of the distinct rules that fired, with the independence groups and the context cap below applied.
3. The tier is significant when a rule marked significant fired or the score reaches thresholds.significant; else suggestive when the score reaches thresholds.suggestive; else noise.

A cluster's tier is not a summed score: it is its strongest member's tier, raised one step only by corroboration between members (FM-14.9).

Independence groups#

Rules that read the same artifact on the same layer are not independent evidence: one fetch program carried through an echo service fires the echo rule, the script rule and the fetch rule at once. Under groups, step 2 becomes: the score is the sum, over groups, of the largest weight among the group's rules that fired, plus the weights of ungrouped rules that fired. significant is unchanged: any significant rule still sets the top tier. The method and the default groups are in FM-4.11.

Context cap#

Rules of context kinds together contribute at most thresholds.suggestive − 1 to a record's score, so a record whose only hits are context kinds stays noise (FM-4.11). A significant context rule still sets the top tier on its own. A group counts as context when the heaviest of its rules that fired is a context kind, a non-context rule winning a tie; its weight then falls under the cap with the ungrouped context rules.

Conformance#

An engine conforms to SC-Rules v1 when, with the rules file at the corpus's rules_version, it gives every vector in conformance/rules-v1.json the same target host, the same set of rule ids, the same score and the same tier. Layer order and numbering are the engine's own. Each vector is a single synthetic URL on a reserved example domain or a public relay or echo service, analysed alone. This repository's own engine is held to the file by a test in sc-core (tests/conformance.rs), which also regenerates the vectors' scores and tiers after an intended scoring change.

The corpus covers each layer type, each tier, the hard negatives the default rules are built to avoid (payload words in a search, a relay tried on the example domain), and the records whose tier the independence groups and the context cap decide:

The full set, with its scores and tiers, is conformance/rules-v1.json, regenerated from the rules at rules_version.

Validating a rules file#

The JSON value of the TOML is validated with any JSON Schema 2020-12 validator. With Python 3.11 or later and the jsonschema package:

python -c "import tomllib, json, jsonschema; jsonschema.validate(tomllib.load(open('rules/default.toml', 'rb')), json.load(open('schemas/sc-rules-v1.json')))"

No output means the file is valid. This repository's gate, scripts/ci/check-rules-schema.py, does the same with the standard library alone (tomllib, json, re), so it needs no third-party package. The engine's own load (swarmchase scan --rules <file>) then checks what the schema cannot: unique ids and patterns that compile.

Versioning#

This is version 1. A minor version adds optional fields or makes a reserved one normative; a file valid under 1.0 stays valid. A major version may change the meaning of a field. A rules file does not declare which spec version it follows; the schema it validates against does.

Source: docs/specs/rules-v1.md