How the bundled rules (version 2026-10-14.5) score on the committed labelled set: 444 labelled records, 239 agent and 205 benign, from the files in samples/eval/. The numbers are computed when the site is built; the same summary is in eval-summary.json.
In plain words
On the 60 labelled records that were collected rather than written for this set (34 from agents, 26 benign):
Suggestive or above. Of 34 agent records, the rules flag 29 at this tier or above (about 85% of them) and miss 5. None of the 29 records they flag is benign. With this few records, the share of flags that are right could be anywhere from 88% to 100%.
Significant or above. Of 34 agent records, the rules flag 14 at this tier or above (about 41% of them) and miss 20. None of the 14 records they flag is benign. With this few records, the share of flags that are right could be anywhere from 78% to 100%.
The rules were tuned on these records, so on new records they do worse. The cards below give each rate as a 95% interval, for these records and for the whole labelled set the build gate holds.
A regression floor, not a precision estimate. The rules were written and tuned against these records, and most of them are synthetic, written to match a rule. The record unit counts every flagged record, so a cluster of ten flagged records counts ten times; the cluster unit and the non-synthetic rows below are the closer guide. Real-world precision is lower. Intervals are 95% Wilson score intervals. See evaluation for the held-out measurement.
Precision and recall by tier
Each rate is given as its 95% interval first, the point estimate under it. The non-synthetic records lead; all records together are the regression floor the gate holds.
suggestive or above
Non-synthetic records (60 labelled: 34 agent, 26 benign)
Precision
88.3–100.0%
95% interval · point 100.0% · 29 of 29 flagged
Recall
69.9–93.6%
95% interval · point 85.3% · 29 of 34 agent records
Regression floor on the labelled set (444 records)
Precision
98.4–100.0%
95% interval · point 100.0% · 234 of 234 flagged
Recall
95.2–99.1%
95% interval · point 97.9% · 234 of 239 agent records
Confusion matrix, suggestive or above, all records
Flagged
Not flagged
Agent
234 true positive
5 false negative
Benign
0 false positive
205 true negative
significant or above
Non-synthetic records (60 labelled: 34 agent, 26 benign)
Precision
78.5–100.0%
95% interval · point 100.0% · 14 of 14 flagged
Recall
26.4–57.8%
95% interval · point 41.2% · 14 of 34 agent records
Regression floor on the labelled set (444 records)
Precision
97.1–100.0%
95% interval · point 100.0% · 127 of 127 flagged
Recall
46.8–59.4%
95% interval · point 53.1% · 127 of 239 agent records
Confusion matrix, significant or above, all records
Flagged
Not flagged
Agent
127 true positive
112 false negative
Benign
0 false positive
205 true negative
Every slice, with its interval
Each point is a rate and each bar its 95% Wilson interval, so a slice with few records shows a wide bar. Synthetic records were written to match the rules; the non-synthetic rows are the closer guide to real data. The record unit counts every flagged record; the cluster unit counts each cluster once.
Precision, record unit
View as table
Slice
Threshold
95% CI
Point
Count
Non-synthetic
suggestive or above
88.3–100.0%
100.0%
29 of 29
Non-synthetic
significant or above
78.5–100.0%
100.0%
14 of 14
All records
suggestive or above
98.4–100.0%
100.0%
234 of 234
All records
significant or above
97.1–100.0%
100.0%
127 of 127
provenance: reconstructed
suggestive or above
88.3–100.0%
100.0%
29 of 29
provenance: reconstructed
significant or above
78.5–100.0%
100.0%
14 of 14
provenance: synthetic
suggestive or above
98.2–100.0%
100.0%
205 of 205
provenance: synthetic
significant or above
96.7–100.0%
100.0%
113 of 113
source: eval
suggestive or above
97.9–100.0%
100.0%
181 of 181
source: eval
significant or above
95.1–100.0%
100.0%
74 of 74
source: urlquery
suggestive or above
51.0–100.0%
100.0%
4 of 4
source: urlquery
significant or above
51.0–100.0%
100.0%
4 of 4
source: urlscan
suggestive or above
86.2–100.0%
100.0%
24 of 24
source: urlscan
significant or above
86.2–100.0%
100.0%
24 of 24
source: wayback
suggestive or above
86.7–100.0%
100.0%
25 of 25
source: wayback
significant or above
86.7–100.0%
100.0%
25 of 25
file: agent-routes.ndjson
suggestive or above
67.6–100.0%
100.0%
8 of 8
file: agent-routes.ndjson
significant or above
51.0–100.0%
100.0%
4 of 4
file: client-traces.ndjson
suggestive or above
74.1–100.0%
100.0%
11 of 11
file: client-traces.ndjson
significant or above
64.6–100.0%
100.0%
7 of 7
file: cross-service.ndjson
suggestive or above
67.6–100.0%
100.0%
8 of 8
file: cross-service.ndjson
significant or above
67.6–100.0%
100.0%
8 of 8
file: linking-negatives.ndjson
suggestive or above
51.0–100.0%
100.0%
4 of 4
file: linking-negatives.ndjson
significant or above
51.0–100.0%
100.0%
4 of 4
file: llm-residue.ndjson
suggestive or above
87.9–100.0%
100.0%
28 of 28
file: llm-residue.ndjson
significant or above
72.2–100.0%
100.0%
10 of 10
file: published-patterns.ndjson
suggestive or above
88.3–100.0%
100.0%
29 of 29
file: published-patterns.ndjson
significant or above
78.5–100.0%
100.0%
14 of 14
file: rule-coverage.ndjson
suggestive or above
94.3–100.0%
100.0%
63 of 63
file: rule-coverage.ndjson
significant or above
85.1–100.0%
100.0%
22 of 22
file: scaffold-fleets.ndjson
suggestive or above
92.1–100.0%
100.0%
45 of 45
file: scaffold-fleets.ndjson
significant or above
92.1–100.0%
100.0%
45 of 45
file: swarm-interaction.ndjson
suggestive or above
90.8–100.0%
100.0%
38 of 38
file: swarm-interaction.ndjson
significant or above
77.2–100.0%
100.0%
13 of 13
Recall, record unit
View as table
Slice
Threshold
95% CI
Point
Count
Non-synthetic
suggestive or above
69.9–93.6%
85.3%
29 of 34
Non-synthetic
significant or above
26.4–57.8%
41.2%
14 of 34
All records
suggestive or above
95.2–99.1%
97.9%
234 of 239
All records
significant or above
46.8–59.4%
53.1%
127 of 239
provenance: reconstructed
suggestive or above
69.9–93.6%
85.3%
29 of 34
provenance: reconstructed
significant or above
26.4–57.8%
41.2%
14 of 34
provenance: synthetic
suggestive or above
98.2–100.0%
100.0%
205 of 205
provenance: synthetic
significant or above
48.3–61.8%
55.1%
113 of 205
source: eval
suggestive or above
93.9–98.8%
97.3%
181 of 186
source: eval
significant or above
33.0–47.0%
39.8%
74 of 186
source: urlquery
suggestive or above
51.0–100.0%
100.0%
4 of 4
source: urlquery
significant or above
51.0–100.0%
100.0%
4 of 4
source: urlscan
suggestive or above
86.2–100.0%
100.0%
24 of 24
source: urlscan
significant or above
86.2–100.0%
100.0%
24 of 24
source: wayback
suggestive or above
86.7–100.0%
100.0%
25 of 25
source: wayback
significant or above
86.7–100.0%
100.0%
25 of 25
file: agent-routes.ndjson
suggestive or above
67.6–100.0%
100.0%
8 of 8
file: agent-routes.ndjson
significant or above
21.5–78.5%
50.0%
4 of 8
file: client-traces.ndjson
suggestive or above
74.1–100.0%
100.0%
11 of 11
file: client-traces.ndjson
significant or above
35.4–84.8%
63.6%
7 of 11
file: cross-service.ndjson
suggestive or above
67.6–100.0%
100.0%
8 of 8
file: cross-service.ndjson
significant or above
67.6–100.0%
100.0%
8 of 8
file: linking-negatives.ndjson
suggestive or above
51.0–100.0%
100.0%
4 of 4
file: linking-negatives.ndjson
significant or above
51.0–100.0%
100.0%
4 of 4
file: llm-residue.ndjson
suggestive or above
87.9–100.0%
100.0%
28 of 28
file: llm-residue.ndjson
significant or above
20.7–54.2%
35.7%
10 of 28
file: published-patterns.ndjson
suggestive or above
69.9–93.6%
85.3%
29 of 34
file: published-patterns.ndjson
significant or above
26.4–57.8%
41.2%
14 of 34
file: rule-coverage.ndjson
suggestive or above
94.3–100.0%
100.0%
63 of 63
file: rule-coverage.ndjson
significant or above
24.3–47.2%
34.9%
22 of 63
file: scaffold-fleets.ndjson
suggestive or above
92.1–100.0%
100.0%
45 of 45
file: scaffold-fleets.ndjson
significant or above
92.1–100.0%
100.0%
45 of 45
file: swarm-interaction.ndjson
suggestive or above
90.8–100.0%
100.0%
38 of 38
file: swarm-interaction.ndjson
significant or above
21.2–50.1%
34.2%
13 of 38
Precision, cluster unit
View as table
Slice
Threshold
95% CI
Point
Count
Non-synthetic
suggestive or above
85.1–100.0%
100.0%
22 of 22
Non-synthetic
significant or above
75.8–100.0%
100.0%
12 of 12
All records
suggestive or above
96.9–100.0%
100.0%
121 of 121
All records
significant or above
94.4–100.0%
100.0%
65 of 65
provenance: reconstructed
suggestive or above
85.1–100.0%
100.0%
22 of 22
provenance: reconstructed
significant or above
75.8–100.0%
100.0%
12 of 12
provenance: synthetic
suggestive or above
96.3–100.0%
100.0%
99 of 99
provenance: synthetic
significant or above
93.2–100.0%
100.0%
53 of 53
source: eval
suggestive or above
96.5–100.0%
100.0%
107 of 107
source: eval
significant or above
93.0–100.0%
100.0%
51 of 51
source: urlquery
suggestive or above
20.7–100.0%
100.0%
1 of 1
source: urlquery
significant or above
20.7–100.0%
100.0%
1 of 1
source: urlscan
suggestive or above
43.9–100.0%
100.0%
3 of 3
source: urlscan
significant or above
43.9–100.0%
100.0%
3 of 3
source: wayback
suggestive or above
74.1–100.0%
100.0%
11 of 11
source: wayback
significant or above
74.1–100.0%
100.0%
11 of 11
file: agent-routes.ndjson
suggestive or above
56.6–100.0%
100.0%
5 of 5
file: agent-routes.ndjson
significant or above
34.2–100.0%
100.0%
2 of 2
file: client-traces.ndjson
suggestive or above
72.2–100.0%
100.0%
10 of 10
file: client-traces.ndjson
significant or above
64.6–100.0%
100.0%
7 of 7
file: cross-service.ndjson
suggestive or above
20.7–100.0%
100.0%
1 of 1
file: cross-service.ndjson
significant or above
20.7–100.0%
100.0%
1 of 1
file: linking-negatives.ndjson
suggestive or above
34.2–100.0%
100.0%
2 of 2
file: linking-negatives.ndjson
significant or above
34.2–100.0%
100.0%
2 of 2
file: llm-residue.ndjson
suggestive or above
84.5–100.0%
100.0%
21 of 21
file: llm-residue.ndjson
significant or above
67.6–100.0%
100.0%
8 of 8
file: published-patterns.ndjson
suggestive or above
85.1–100.0%
100.0%
22 of 22
file: published-patterns.ndjson
significant or above
75.8–100.0%
100.0%
12 of 12
file: rule-coverage.ndjson
suggestive or above
91.2–100.0%
100.0%
40 of 40
file: rule-coverage.ndjson
significant or above
81.6–100.0%
100.0%
17 of 17
file: scaffold-fleets.ndjson
suggestive or above
77.2–100.0%
100.0%
13 of 13
file: scaffold-fleets.ndjson
significant or above
77.2–100.0%
100.0%
13 of 13
file: swarm-interaction.ndjson
suggestive or above
64.6–100.0%
100.0%
7 of 7
file: swarm-interaction.ndjson
significant or above
43.9–100.0%
100.0%
3 of 3
Recall, cluster unit
View as table
Slice
Threshold
95% CI
Point
Count
Non-synthetic
suggestive or above
66.5–93.8%
84.6%
22 of 26
Non-synthetic
significant or above
28.8–64.5%
46.2%
12 of 26
All records
suggestive or above
92.1–98.7%
96.8%
121 of 125
All records
significant or above
43.3–60.6%
52.0%
65 of 125
provenance: reconstructed
suggestive or above
66.5–93.8%
84.6%
22 of 26
provenance: reconstructed
significant or above
28.8–64.5%
46.2%
12 of 26
provenance: synthetic
suggestive or above
96.3–100.0%
100.0%
99 of 99
provenance: synthetic
significant or above
43.8–63.0%
53.5%
53 of 99
source: eval
suggestive or above
91.1–98.6%
96.4%
107 of 111
source: eval
significant or above
37.0–55.2%
45.9%
51 of 111
source: urlquery
suggestive or above
20.7–100.0%
100.0%
1 of 1
source: urlquery
significant or above
20.7–100.0%
100.0%
1 of 1
source: urlscan
suggestive or above
43.9–100.0%
100.0%
3 of 3
source: urlscan
significant or above
43.9–100.0%
100.0%
3 of 3
source: wayback
suggestive or above
74.1–100.0%
100.0%
11 of 11
source: wayback
significant or above
74.1–100.0%
100.0%
11 of 11
file: agent-routes.ndjson
suggestive or above
56.6–100.0%
100.0%
5 of 5
file: agent-routes.ndjson
significant or above
11.8–76.9%
40.0%
2 of 5
file: client-traces.ndjson
suggestive or above
72.2–100.0%
100.0%
10 of 10
file: client-traces.ndjson
significant or above
39.7–89.2%
70.0%
7 of 10
file: cross-service.ndjson
suggestive or above
20.7–100.0%
100.0%
1 of 1
file: cross-service.ndjson
significant or above
20.7–100.0%
100.0%
1 of 1
file: linking-negatives.ndjson
suggestive or above
34.2–100.0%
100.0%
2 of 2
file: linking-negatives.ndjson
significant or above
34.2–100.0%
100.0%
2 of 2
file: llm-residue.ndjson
suggestive or above
84.5–100.0%
100.0%
21 of 21
file: llm-residue.ndjson
significant or above
20.8–59.1%
38.1%
8 of 21
file: published-patterns.ndjson
suggestive or above
66.5–93.8%
84.6%
22 of 26
file: published-patterns.ndjson
significant or above
28.8–64.5%
46.2%
12 of 26
file: rule-coverage.ndjson
suggestive or above
91.2–100.0%
100.0%
40 of 40
file: rule-coverage.ndjson
significant or above
28.5–57.8%
42.5%
17 of 40
file: scaffold-fleets.ndjson
suggestive or above
77.2–100.0%
100.0%
13 of 13
file: scaffold-fleets.ndjson
significant or above
77.2–100.0%
100.0%
13 of 13
file: swarm-interaction.ndjson
suggestive or above
64.6–100.0%
100.0%
7 of 7
file: swarm-interaction.ndjson
significant or above
15.8–75.0%
42.9%
3 of 7
Precision at each record tier
The share of agent records among records at exactly this tier: how far each step can be trusted.
Precision at each agent grade
The share of agent records among records at exactly this grade (records graded by their cluster; unmeasured on held-out data): how far each step can be trusted.
Hits per rule
Labelled records each rule matched, agent then benign, with the rule's own precision once it has 5 hits or more. A rule with benign hits is a candidate for a lower weight. Coverage: 103 of 103 rules have an agent example.
Show the chart of hits per rule
agent recordsbenign records
View as table (every rule)
Rule
Tactic
Agent
Benign
Precision
Hard negatives fired
Agent example
relay.jina
T1 task retrieval
70
13
84.3% 75.0–90.6%
6/6
yes
relay.jina_search
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.repo_digest
T1 task retrieval
1
2
33.3% 6.1–79.2%
2/3
yes
relay.markdown_new
T1 task retrieval
3
1
75.0% 30.1–95.4%
1/1
yes
relay.microlink
T1 task retrieval
5
2
71.4% 35.9–91.8%
1/1
yes
relay.hrefli
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.translate_goog
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.allorigins
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.corsproxy
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.pure_md
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.defuddle_md
T1 task retrieval
1
1
50.0% 9.5–90.5%
1/1
yes
relay.archive_raw
T1 task retrieval
1
0
100.0% 20.7–100.0%
0/5
yes
relay.scrape_api
T1 task retrieval
2
2
50.0% 15.0–85.0%
2/2
yes
carrier.httpbin
T2 tool escalation
10
8
55.6% 33.7–75.4%
7/7
yes
carrier.livecodes
T2 tool escalation
1
1
50.0% 9.5–90.5%
1/1
yes
carrier.codesandbox
T2 tool escalation
1
1
50.0% 9.5–90.5%
1/1
yes
carrier.browserless
T2 tool escalation
1
1
50.0% 9.5–90.5%
1/1
yes
carrier.remote_browser_cdp
T2 tool escalation
2
1
66.7% 20.8–93.9%
1/2
yes
carrier.agent_sandbox
T2 tool escalation
1
1
50.0% 9.5–90.5%
1/1
yes
carrier.embed_playground
T2 tool escalation
1
1
50.0% 9.5–90.5%
1/1
yes
carrier.microlink_function
T2 tool escalation
1
0
100.0% 20.7–100.0%
0/1
yes
carrier.data_url_script
T2 tool escalation
6
0
100.0% 61.0–100.0%
0/2
yes
carrier.inline_fetch_program
T2 tool escalation
9
0
100.0% 70.1–100.0%
0/2
yes
carrier.form_post_program
T2 tool escalation
5
0
100.0% 56.6–100.0%
0/2
yes
carrier.html_program
T2 tool escalation
15
4
78.9% 56.7–91.5%
4/4
yes
exploit.sqli_union
T4 vulnerability probing
5
0
100.0% 56.6–100.0%
0/5
yes
exploit.sqli_boolean
T4 vulnerability probing
65
0
100.0% 94.4–100.0%
0/1
yes
exploit.traversal
T4 vulnerability probing
6
0
100.0% 61.0–100.0%
0/2
yes
exploit.traversal_single
T4 vulnerability probing
7
0
100.0% 64.6–100.0%
0/2
yes
exploit.sqli_quote
T4 vulnerability probing
21
0
100.0% 84.5–100.0%
0/1
yes
exploit.xss
T4 vulnerability probing
17
0
100.0% 81.6–100.0%
0/1
yes
exploit.ssti
T4 vulnerability probing
1
0
100.0% 20.7–100.0%
0/1
yes
exploit.cmdi
T4 vulnerability probing
2
0
100.0% 34.2–100.0%
0/2
yes
exploit.boundary_ids
T4 vulnerability probing
1
1
50.0% 9.5–90.5%
1/1
yes
exploit.cmd_param
T4 vulnerability probing
1
0
100.0% 20.7–100.0%
0/3
yes
exploit.open_redirect_probe
T4 vulnerability probing
1
0
100.0% 20.7–100.0%
0/1
yes
exploit.null_byte
T4 vulnerability probing
1
0
100.0% 20.7–100.0%
0/1
yes
exploit.userinfo_confusion
T4 vulnerability probing
1
0
100.0% 20.7–100.0%
0/1
yes
exploit.php_eval
T4 vulnerability probing
1
0
100.0% 20.7–100.0%
0/1
yes
marker.uq_tags
untagged
109
0
100.0% 96.6–100.0%
0/3
yes
marker.test_tag
untagged
2
0
100.0% 34.2–100.0%
0/1
yes
marker.probe_tag
untagged
2
0
100.0% 34.2–100.0%
0/2
yes
marker.oai_tag
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
marker.callback_seq
T7 callback or side channels
1
0
100.0% 20.7–100.0%
0/1
yes
marker.title_write
T7 callback or side channels
2
1
66.7% 20.8–93.9%
1/1
yes
marker.md_link
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
marker.md_url_wrapper
untagged
1
0
100.0% 20.7–100.0%
0/3
yes
marker.placeholder
untagged
14
0
100.0% 78.5–100.0%
0/1
yes
marker.template_unrendered
untagged
1
2
33.3% 6.1–79.2%
1/1
yes
marker.example_host_value
untagged
1
1
50.0% 9.5–90.5%
1/2
yes
marker.tool_call_markup
untagged
2
0
100.0% 34.2–100.0%
0/1
yes
marker.tool_call_json
untagged
2
0
100.0% 34.2–100.0%
0/3
yes
marker.json_quote_residue
untagged
2
0
100.0% 34.2–100.0%
0/3
yes
marker.mcp_call
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
marker.mcp_tool_name
untagged
1
1
50.0% 9.5–90.5%
1/1
yes
marker.agent_discovery
untagged
2
0
100.0% 34.2–100.0%
0/1
yes
marker.escaped_newline
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
marker.truncated_output
untagged
2
0
100.0% 34.2–100.0%
0/3
yes
marker.chat_opener
untagged
4
0
100.0% 51.0–100.0%
0/4
yes
marker.doubled_scheme
untagged
2
0
100.0% 34.2–100.0%
0/2
yes
marker.garbled_scheme
untagged
2
0
100.0% 34.2–100.0%
0/2
yes
marker.impossible_date
untagged
3
0
100.0% 43.9–100.0%
0/2
yes
marker.placeholder_identity
T5 account or identity acquisition
2
3
40.0% 11.8–76.9%
3/3
yes
marker.repeated_token
untagged
2
0
100.0% 34.2–100.0%
0/4
yes
cache.buster
T1 task retrieval
41
5
89.1% 77.0–95.3%
4/5
yes
selfid.vendor_name
untagged
5
8
38.5% 17.7–64.5%
4/6
yes
exfil.webhook_site
T7 callback or side channels
8
0
100.0% 67.6–100.0%
0/2
yes
exfil.requestbin
T7 callback or side channels
3
0
100.0% 43.9–100.0%
0/1
yes
exfil.collaborator
T7 callback or side channels
1
0
100.0% 20.7–100.0%
0/1
yes
exfil.dnslog
T7 callback or side channels
1
0
100.0% 20.7–100.0%
0/1
yes
exfil.oast_service
T7 callback or side channels
1
0
100.0% 20.7–100.0%
0/2
yes
exfil.ntfy
T7 callback or side channels
2
0
100.0% 34.2–100.0%
0/1
yes
exfil.tunnel
T7 callback or side channels
1
1
50.0% 9.5–90.5%
1/1
yes
exfil.paste
T7 callback or side channels
2
1
66.7% 20.8–93.9%
1/1
yes
exfil.cookie
T6 credential reuse
1
0
100.0% 20.7–100.0%
0/1
yes
exfil.disposable_mail
T5 account or identity acquisition
2
1
66.7% 20.8–93.9%
1/1
yes
antibot.baxia
T3 access-control circumvention
1
0
100.0% 20.7–100.0%
0/1
yes
antibot.captcha_solver
T3 access-control circumvention
1
0
100.0% 20.7–100.0%
0/1
yes
antibot.ocr_captcha
T3 access-control circumvention
1
0
100.0% 20.7–100.0%
0/1
yes
antibot.unblocker_params
T3 access-control circumvention
1
1
50.0% 9.5–90.5%
1/1
yes
header.via_hysandbox
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
header.via_ats
untagged
2
2
50.0% 15.0–85.0%
1/1
yes
header.gzip_only_requests
untagged
1
2
33.3% 6.1–79.2%
1/1
yes
header.scripted_ua
untagged
14
5
73.7% 51.2–88.2%
3/4
yes
header.headless_ua
untagged
5
1
83.3% 43.6–97.0%
1/1
yes
header.mcp_fetch_ua
untagged
1
1
50.0% 9.5–90.5%
1/1
yes
header.cf_browser_rendering
untagged
2
1
66.7% 20.8–93.9%
1/1
yes
header.ua_platform_mismatch
untagged
1
1
50.0% 9.5–90.5%
1/2
yes
header.pentest_ua
untagged
1
5
16.7% 3.0–56.4%
5/5
yes
outcome.neterror
T3 access-control circumvention
1
1
50.0% 9.5–90.5%
1/1
yes
outcome.challenge
T3 access-control circumvention
1
1
50.0% 9.5–90.5%
1/1
yes
infra.hysandbox_host
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
exploit.double_encoded_path
T4 vulnerability probing
3
0
100.0% 43.9–100.0%
0/1
yes
exploit.double_encoded_meta
T4 vulnerability probing
5
0
100.0% 56.6–100.0%
0/3
yes
antibot.incapsula
T3 access-control circumvention
1
0
100.0% 20.7–100.0%
0/1
yes
exfil.httpbin_data
T7 callback or side channels
3
0
100.0% 43.9–100.0%
0/1
yes
marker.oai_prefix
untagged
1
0
100.0% 20.7–100.0%
0/1
yes
selfid.signature_agent
untagged
2
3
40.0% 11.8–76.9%
1/1
yes
header.agent_ua
untagged
4
3
57.1% 25.0–84.2%
2/3
yes
header.markdown_accept
untagged
1
1
50.0% 9.5–90.5%
1/1
yes
selfid.agent_commerce
untagged
1
2
33.3% 6.1–79.2%
2/2
yes
header.coding_agent_ua
untagged
6
4
60.0% 31.3–83.2%
4/5
yes
marker.assistant_citation
untagged
2
6
25.0% 7.1–59.1%
5/6
yes
Recall by class
Agent records of each labelled class, by the tier the rules gave them. A class the rules cannot see from one record shows as missed rather than dragging down one averaged number.
significantsuggestivemissed (noise)
View as table
Name
Agent records
Recall, suggestive or above
Recall, significant
agent_marker
10
100.0% 72.2–100.0%
10.0% 1.8–40.4%
anti_bot
5
100.0% 56.6–100.0%
40.0% 11.8–76.9%
coordination
38
100.0% 90.8–100.0%
34.2% 21.2–50.1%
custom_program
16
100.0% 80.6–100.0%
25.0% 10.2–49.5%
exfil
16
100.0% 80.6–100.0%
25.0% 10.2–49.5%
exploit
93
98.9% 94.2–99.8%
92.5% 85.3–96.3%
fuzzing
2
0.0% 0.0–65.8%
0.0% 0.0–65.8%
indirection
17
88.2% 65.7–96.7%
29.4% 13.3–53.1%
infrastructure
13
100.0% 77.2–100.0%
15.4% 4.3–42.2%
llm_residue
29
100.0% 88.3–100.0%
34.5% 19.9–52.7%
Recall by tactic
Agent records grouped by the tactic of the rules that fired on them (a record counts under each tactic it touches). It shows how strongly each tactic's evidence scores; what the rules miss is under no rule fired.
significantsuggestivemissed (noise)
View as table
Name
Agent records
Recall, suggestive or above
Recall, significant
T1 task retrieval
95
97.9% 92.6–99.4%
40.0% 30.7–50.1%
T2 tool escalation
29
100.0% 88.3–100.0%
51.7% 34.4–68.6%
T3 access-control circumvention
7
100.0% 64.6–100.0%
28.6% 8.2–64.1%
T4 vulnerability probing
129
99.2% 95.7–99.9%
84.5% 77.3–89.7%
T5 account or identity acquisition
4
100.0% 51.0–100.0%
25.0% 4.6–69.9%
T6 credential reuse
1
100.0% 20.7–100.0%
100.0% 20.7–100.0%
T7 callback or side channels
25
100.0% 86.7–100.0%
44.0% 26.7–62.9%
no rule fired
2
0.0% 0.0–65.8%
0.0% 0.0–65.8%
untagged rules
161
100.0% 97.7–100.0%
55.9% 48.2–63.3%
False positives by benign class
Benign records of each class the rules flagged, with the rate and its interval. Hard negatives are benign records written to sit next to a rule's pattern.
Class
Records
Flagged suggestive or above
Flagged significant
ai_referral
5
0 0.0% 0.0–43.4%
0 0.0% 0.0–43.4%
api_client
22
0 0.0% 0.0–14.9%
0 0.0% 0.0–14.9%
archive_link
7
0 0.0% 0.0–35.4%
0 0.0% 0.0–35.4%
crawler
22
0 0.0% 0.0–14.9%
0 0.0% 0.0–14.9%
dev_tool
39
0 0.0% 0.0–9.0%
0 0.0% 0.0–9.0%
human_browse
35
0 0.0% 0.0–9.9%
0 0.0% 0.0–9.9%
human_form
40
0 0.0% 0.0–8.8%
0 0.0% 0.0–8.8%
mail_gateway
10
0 0.0% 0.0–27.8%
0 0.0% 0.0–27.8%
relay_reading
20
0 0.0% 0.0–16.1%
0 0.0% 0.0–16.1%
scanner
5
0 0.0% 0.0–43.4%
0 0.0% 0.0–43.4%
Linking
Records labelled with the run they belong to measure how well clusters recover runs. A merge of runs inside one labelled swarm counts as right for precision, and is not a false merge. The eval gate holds a floor on B-cubed F1.
B-cubed precision
100.0%
B-cubed recall
83.3%
B-cubed F1
90.9%
False merges
0.0%
0 of 50 clusters with labelled members
183 run-labelled records in 61 runs. Runs split across clusters: cite-run-1, cli-run-aider, cli-run-gemini, cov-run-a, cov-run-d, cov-run-g, cov-run-h, cov-run-j, cov-run-l, pub-bea, pub-calaccess, pub-ed, pub-unctad, pub-unm.
Recompute with edited rules
Scores the bundled rules and the rules edited on the rules page (kept in this browser) side by side, in this page, and lists the records whose tier changed. Nothing leaves the browser.