feat(examples): add ATT&CK threat report mapper

examples/README.md

@@ -32,6 +32,7 @@ service keys.
 | [Reproduce CMS's L1851 documentation finding](./prior-authorization-review-agent) | Tracing a published six-month requirement against a seven-month face-to-face encounter and CMS's recoupment result | `extract`, `encode`, `score` | SIE endpoint; standalone `uv` project; exact CMS published example | Runnable agent example |
 | [Reconstruct a bearing failure](./maintenance-triage-agent) | Turning the NTSB's three East Palestine detector readings into a cited temperature and alert sequence without adding a new causal claim | `extract`, `encode`, `score` | SIE endpoint; standalone `uv` project; exact NTSB illustrated report spread | Runnable agent example |
 | [Make a shelf gap auditable](./retail-shelf-audit) | Detecting one empty facing, deriving its notice and shelf-label crops by geometry, then preserving OCR evidence | `extract` | GPU SIE deployment; standalone `uv` project; CC0 supermarket shelf image and recorded direct-checkpoint evidence included | Runnable evaluation example |
+| [Turn threat reports into cited ATT&CK mapping suggestions](./threat-report-attck-mapper) | Mapping full reports against active ATT&CK 19.2, with a separate pinned AnnoCTR linking benchmark and analyst review for every suggestion | `generate`, `extract`, `encode`, `score` | GPU SIE deployment; standalone `uv` project; pinned MITRE ATT&CK and AnnoCTR sources | Runnable agent benchmark |
 | [A behavioural gate that catches hijacked AI agents by their actions, not their credentials](./agent-action-monitor) | Judging a proposed AI agent action against that agent's own learned baseline in real time, before it reaches a downstream system | `encode`, `score`, `extract` | Docker Compose (gate + self-hosted SIE + n8n + mock downstream), no API key required | Runnable demo |
 
 For docs publishing, lead with the quickest runnable demos, then use the

examples/threat-report-attck-mapper/.env.example

@@ -0,0 +1,2 @@
+SIE_CLUSTER_URL=http://localhost:8080
+SIE_API_KEY=

examples/threat-report-attck-mapper/.gitignore

@@ -0,0 +1,8 @@
+.env
+.venv/
+.pytest_cache/
+.ruff_cache/
+__pycache__/
+data/cache/
+runs/*
+!runs/.gitkeep

examples/threat-report-attck-mapper/EXPERIMENT.md

@@ -0,0 +1,88 @@
+# End-to-end ATT&CK mapping experiment
+
+## Development result
+
+The agent starts with complete report text. It extracted a behavior overlapping
+92 of 128 known report-technique pairs across 16 AnnoCTR development reports.
+The candidate ensemble kept the correct technique family for 84 of those 92
+pairs.
+
+The 27B verifier and nearest labeled training example agreed on 50 suggestions
+that overlap an AnnoCTR annotation. Forty-four used the exact reference ID, for
+88.0% selective precision. Those agreements become direct suggestions. Other
+27B-supported mappings stay in the analyst review queue with the same source
+quote and candidate ledger.
+
+AnnoCTR has no annotation beside 48 other direct suggestions. The evaluator
+does not assume those suggestions are correct. At the report-pair level, 29 of
+54 direct suggestions match the published reference. That 53.7% reference
+match rate remains in the run artifact.
+
+Every one of the 351 extracted behaviors retains exact global character
+offsets. The worked Proofpoint report is absent from the aggregate.
+
+## Why the output changed
+
+The first development hypothesis treated every annotated noun occurrence as a
+separate target. It reached 4.7% implicit-mention recall and 12.5% precision.
+That unit rewarded reproducing repeated words such as `ransomware` instead of
+preparing a report-level review queue.
+
+The next version emitted every mapping accepted by the 27B verifier. It found
+more of the reference, then sent 201 unique report-technique pairs to review.
+Only 41 matched an annotated span and ID. The verifier was acting as a mapper,
+with almost no abstention.
+
+The final development revision adds 1,915 labeled ATT&CK spans from the
+AnnoCTR training split as an independent retrieval path. Dense definitions,
+token-level MaxSim, and labeled examples form one candidate pool. A direct
+suggestion requires agreement between the nearest labeled example and the 27B
+verifier. Disagreements remain visible for analyst review.
+
+## Frozen held-out result
+
+The fixed pipeline ran once on the 33-report AnnoCTR test split on August 21,
+2026. It saw 317 exact report-technique pairs after the frozen exclusions.
+
+| Release check | Frozen result | Gate | Outcome |
+|---|---:|---:|---|
+| Direct-suggestion precision on annotated spans | 91/109 (83.5%) | 85% | Missed |
+| Behavior extraction recall over report-technique pairs | 199/317 (62.8%) | 70% | Missed |
+| Family finalist recall after behavior extraction | 180/199 (90.5%) | 90% | Passed |
+| Exact source offsets | Passed | Required | Passed |
+| Worked report absent from the aggregate | Passed | Required | Passed |
+
+The aggregate did not clear its release gate. The held-out precision result was
+4.5 points below development, while behavior recall fell 9.1 points. Candidate
+retrieval held: once extraction found the behavior, the correct ATT&CK family
+reached the finalist set in 90.5% of report-technique pairs.
+
+No prompt, threshold, rank weight, candidate count, or routing rule changed
+after this result. The worked Proofpoint report remains a separate case study;
+it is not evidence that the aggregate passed.
+
+## Dataset boundary
+
+- Source: AnnoCTR commit `d510b6949e1938d47c93a43eedd562dc538439dc`.
+- Labeled retrieval examples come from the published train split.
+- Development uses the published dev split.
+- Final measurement used the published test split once on August 21, 2026.
+- `proofpoint_2022-02-03_mfa-psa-oh-my` is excluded from the aggregate because
+  it informed the example design. It remains the worked report.
+
+The aggregate uses the ATT&CK catalog bundled with AnnoCTR. The worked report
+uses ATT&CK Enterprise 19.2. AnnoCTR's labels are a reproducible reference;
+they are not an expert-consensus claim.
+
+## Prediction and matching contract
+
+A behavior contains an exact quote, global offsets, typed event fields, and a
+candidate ledger. A direct suggestion adds one ATT&CK ID whose 27B selection
+matches the top labeled-example technique. Every output still requires analyst
+acceptance.
+
+Gold mentions collapse to unique `(report, technique)` pairs. Direct
+suggestions collapse the same way. A pair matches when the report and exact
+technique ID match and at least one cited span overlaps a reference mention.
+Parent and sub-technique IDs must match for the exact score. Family scores stay
+separate.

examples/threat-report-attck-mapper/README.md

@@ -0,0 +1,145 @@
+# Turn complete threat reports into cited ATT&CK suggestions
+
+This agent starts with the report text. It finds adversary behavior, resolves
+the closest MITRE ATT&CK technique, and keeps the exact source passage beside
+every suggestion. No annotated behavior spans are supplied at runtime.
+
+Every mapping remains `needs_analyst_review`. The agent can suggest a technique
+or abstain; it cannot accept a mapping.
+
+## Model ensemble
+
+| Job | Model | SIE primitive |
+|---|---|---|
+| Extract typed adversary events | `Qwen/Qwen3.6-27B:no-spec` | `generate` with a strict JSON schema |
+| Tag actors, tools, credentials, and targets | `fastino/gliner2-large-v1` | `extract` |
+| Retrieve ATT&CK definitions with one vector per text | `Qwen/Qwen3-Embedding-8B` | `encode` |
+| Retrieve labeled report spans from AnnoCTR train | `Qwen/Qwen3-Embedding-8B` | `encode` |
+| Retrieve ATT&CK definitions with one vector per token | `jinaai/jina-colbert-v2` | `encode` with `multivector` output |
+| Rerank the joint candidate set | `Qwen/Qwen3-Reranker-4B` | `score` |
+| Verify the technique against the quoted evidence | `Qwen/Qwen3.6-27B:no-spec` | `generate` with a strict JSON schema |
+
+Dense retrieval and late interaction each search the complete ATT&CK catalog.
+A third search compares the extracted behavior with labeled spans from the
+AnnoCTR training split, then keeps the nearest example for each technique. The
+agent fuses two top-50 definition pools with 20 example-backed candidates and
+sends 75 candidates to the reranker. The verifier reads the best ten, including
+the nearest labeled source example when one reached the pool.
+
+A verified mapping becomes a direct suggestion when the selected technique is
+also the top labeled-example match. Other verified mappings stay in the
+analyst review queue. Both routes keep the quote, event fields, and candidate
+ledger.
+
+Late interaction uses one query vector per token against every ATT&CK document
+multivector. MaxSim scores each query token against its closest document token,
+then sums those matches. No query pooling happens before candidate fusion.
+
+`config.yaml` pins every model revision. A managed SIE run also records request
+IDs, the settled rate-book version, and execution-identity hashes.
+
+Behavior extraction bounds the JSON array in the grammar for each report
+chunk. If a response still ends before valid JSON closes, the retry splits the
+text and halves the row limit. The failed response remains in the API ledger.
+
+## Data
+
+`attck-map fetch` downloads two pinned sources:
+
+- MITRE ATT&CK Enterprise 19.2, with 697 active techniques in the August 5,
+  2026 STIX bundle.
+- AnnoCTR at commit `d510b694`, including its published train, dev, and test
+  splits plus the corpus's 578-technique ATT&CK snapshot under CC BY-SA 4.0.
+
+The agent indexes labeled technique spans from AnnoCTR train. Development and
+test reports never enter that index. The full-report evaluation still begins
+with raw report text; it supplies no target spans or labels for those reports.
+
+The command verifies both SHA-256 hashes. See `fixtures/SOURCES.md` for source
+and license details.
+
+## Run the agent
+
+You need Python 3.12 and a SIE endpoint that serves the models in `config.yaml`.
+The example sends every model call to `SIE_CLUSTER_URL`.
+
+```bash
+cd examples/threat-report-attck-mapper
+cp .env.example .env
+uv sync --frozen
+
+uv run attck-map fetch
+uv run attck-map report path/to/report.pdf --run-id report-review
+uv run attck-map demo --run-id mfa-psa
+```
+
+The copied `.env` points to `http://localhost:8080`. Set `SIE_CLUSTER_URL` and
+`SIE_API_KEY` to use SIE Cloud.
+
+`demo` reads Proofpoint's “MFA PSA, Oh My!” report from AnnoCTR. The report
+describes reverse-proxy phishing without supplying the current ATT&CK mapping.
+The agent must separate interception, cookie theft, and reuse of a stolen
+cookie, including current sub-technique `T1550.004`.
+
+The checked-in run selected `T1550.004` for the exact words “use the stolen
+session cookie to log in as the victim.” Its nearest labeled AnnoCTR example
+pointed to `T1539`, which describes stealing the cookie. The routing policy kept
+the reuse mapping and placed it in the closer-review lane. See `verified-run/`
+for the report, raw SIE calls, candidate ledger, and checksummed manifest.
+
+## Run the raw-report evaluation
+
+The end-to-end benchmark starts from every complete report in a published
+AnnoCTR split:
+
+```bash
+uv run attck-map full-benchmark --split dev --run-id dev-full
+uv run attck-map full-benchmark --split test --run-id test-frozen
+```
+
+Use development data while changing prompts or thresholds. `EXPERIMENT.md`
+freezes the held-out test protocol and excludes the Proofpoint worked example
+from the aggregate.
+
+Each run writes the extracted event spans, candidate ledgers, final mappings,
+gold mentions, metrics, API calls, and a manifest with artifact hashes. The API
+ledger keeps the exact payload and raw response for extraction, reranking, and
+verification. Dense and token-level vectors stay in compressed NumPy files.
+The evaluation reports:
+
+- direct-suggestion precision where AnnoCTR annotates the cited span;
+- behavior and candidate recall from complete reports;
+- report-pair reference matches as a coverage diagnostic;
+- candidate contribution from dense retrieval, MaxSim, labeled examples, and
+  their overlaps.
+
+The older `benchmark` command starts from human-annotated spans. Keep it for
+retrieval diagnostics; it is not the end-to-end result.
+
+## Frozen held-out result
+
+The fixed pipeline ran once on the 33-report AnnoCTR test split on August 21,
+2026. The Proofpoint worked report was excluded before the run.
+
+| Release check | Test result | Gate |
+|---|---:|---:|
+| Exact ATT&CK ID when a direct suggestion overlaps an annotated span | 91/109 (83.5%) | 85% |
+| Report-technique pairs with an extracted behavior | 199/317 (62.8%) | 70% |
+| Correct technique family reached the finalist set after extraction | 180/199 (90.5%) | 90% |
+| Exact source offsets | Passed | Required |
+| Worked report absent from the aggregate | Passed | Required |
+
+The family finalist gate passed. The precision and behavior-recall gates did
+not. No prompt, threshold, or rank weight was changed after seeing the test
+result. `EXPERIMENT.md` records the development result and the frozen test
+contract.
+
+## Decision boundary
+
+This project prepares ATT&CK suggestions for human review. It does not detect
+an intrusion, change a security control, or claim that a report author endorsed
+the suggested mapping.
+
+MITRE ATT&CK is a registered trademark of The MITRE Corporation. AnnoCTR report
+text and annotations are licensed CC BY-SA 4.0 by the corpus authors and source
+contributors.

examples/threat-report-attck-mapper/config.yaml

@@ -0,0 +1,67 @@
+cluster:
+  url: "http://localhost:8080"
+  api_key: ""
+  provision_timeout_s: 900
+
+models:
+  parse: "docling"
+  behavior_extract: "Qwen/Qwen3.6-27B:no-spec"
+  entities: "fastino/gliner2-large-v1"
+  retrieve: "Qwen/Qwen3-Embedding-8B"
+  late_interaction: "jinaai/jina-colbert-v2"
+  rerank: "Qwen/Qwen3-Reranker-4B"
+  verify: "Qwen/Qwen3.6-27B:no-spec"
+  escalate: "Qwen/Qwen3.6-27B:no-spec"
+
+model_revisions:
+  docling: "310479be56f3a99a0fea75d930e11eea38bf5ef7"
+  fastino/gliner2-large-v1: "b122b11eeaee4dabd32bed80412f3234c0d0e943"
+  Qwen/Qwen3-Embedding-8B: "1d8ad4ca9b3dd8059ad90a75d4983776a23d44af"
+  jinaai/jina-colbert-v2: "4552c4dc1ffd7d7a635b6a41a1077fe9c9cdd974"
+  Qwen/Qwen3-Reranker-4B: "22e683669bc0f0bd69640a1354a6d0aebcfeede5"
+  Qwen/Qwen3.6-27B:no-spec: "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9"
+
+retrieval:
+  instruction: "Given a behavior described in a cyber threat report, retrieve the MITRE ATT&CK Enterprise technique that directly explains the behavior."
+  exemplar_instruction: "Represent this threat-report span for ATT&CK technique classification."
+  candidate_count: 75
+  dense_pool_count: 50
+  late_interaction_pool_count: 50
+  exemplar_pool_count: 20
+  exemplar_rrf_weight: 2.0
+  rerank_count: 75
+  verifier_count: 10
+  embedding_batch_size: 16
+  multivector_batch_size: 16
+
+report:
+  max_behaviors: 24
+  # Keeps the extraction prompt plus a 4096-token JSON response inside the
+  # 8192-token Qwen3.6 serving profile, including the 40 entity hints.
+  chunk_characters: 4500
+  entity_chunk_characters: 2000
+  chunk_overlap_characters: 400
+  use_escalation: false
+
+evaluation:
+  suggestion_annotated_span_precision_gate: 0.85
+  report_pair_behavior_recall_gate: 0.70
+  report_pair_family_conditional_finalist_recall_gate: 0.90
+  excluded_documents:
+    - "proofpoint_2022-02-03_mfa-psa-oh-my"
+
+concurrency:
+  documents: 8
+  rerank: 16
+  verify: 8
+
+sources:
+  attack:
+    version: "19.2"
+    commit: "6cda5ad8462c79e14fbb872f4e09059b18e0cfc4"
+    url: "https://raw.githubusercontent.com/mitre-attack/attack-stix-data/6cda5ad8462c79e14fbb872f4e09059b18e0cfc4/enterprise-attack/enterprise-attack-19.2.json"
+    sha256: "dc1639caa5501d720e280cf1cbd8fbe009884a0c9b3e6e9ed9d0c25166c3d8f4"
+  annoctr:
+    commit: "d510b6949e1938d47c93a43eedd562dc538439dc"
+    url: "https://github.com/boschresearch/anno-ctr-lrec-coling-2024/archive/d510b6949e1938d47c93a43eedd562dc538439dc.zip"
+    sha256: "2eb29db3bba2e95f908ad3d4195c202b3c44a666f98c43e4c02a809edffcd480"

examples/threat-report-attck-mapper/fixtures/SOURCES.md

@@ -0,0 +1,36 @@
+# Sources and evaluation boundary
+
+The example downloads two immutable public sources and verifies each file by
+SHA-256 before use.
+
+## MITRE ATT&CK Enterprise 19.2
+
+- Source: `mitre-attack/attack-stix-data`
+- Release: Enterprise ATT&CK 19.2, modified August 5, 2026
+- Commit: `6cda5ad8462c79e14fbb872f4e09059b18e0cfc4`
+- Terms: MITRE ATT&CK terms of use. ATT&CK must be attributed to The MITRE
+  Corporation.
+
+## AnnoCTR
+
+- Source: `boschresearch/anno-ctr-lrec-coling-2024`
+- Commit: `d510b6949e1938d47c93a43eedd562dc538439dc`
+- License: [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) for
+  the `AnnoCTR/` corpus
+- Paper: Lukas Lange et al., [“AnnoCTR: A Dataset for Detecting and Linking
+  Entities, Tactics, and Techniques in Cyber Threat
+  Reports”](https://aclanthology.org/2024.lrec-main.103/), LREC-COLING 2024.
+
+Proofpoint contributed “MFA PSA, Oh My!” to AnnoCTR and remains the original
+publisher of the complete report used in `verified-run/parsed-report.md`.
+
+The benchmark uses AnnoCTR's published train, dev, and test files. Rows that
+share one annotated span are grouped into a single multi-label case. Historical
+linking is scored against the 578-technique MITRE entity snapshot distributed
+with AnnoCTR. Full-report review uses active ATT&CK 19.2. Keeping the catalogs
+separate prevents a correct current mapping from being marked wrong because an
+older technique definition or label changed.
+
+The legacy `benchmark` command starts from an annotated behavior span. The
+`full-benchmark` command starts from complete report text and evaluates behavior
+extraction separately from ATT&CK linking.

examples/threat-report-attck-mapper/pyproject.toml

@@ -0,0 +1,37 @@
+[project]
+name = "threat-report-attck-mapper"
+version = "0.1.0"
+description = "Map cited threat-report behavior to MITRE ATT&CK with SIE"
+readme = "README.md"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    "httpx>=0.27",
+    "numpy>=1.26",
+    "python-dotenv>=1.0",
+    "pyyaml>=6.0",
+    "rich>=13.7",
+    "sie-sdk==0.7.1",
+]
+
+[project.scripts]
+attck-map = "threat_mapper.cli:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0",
+    "ruff>=0.14",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["threat_mapper"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py312"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

examples/threat-report-attck-mapper/runs/.gitkeep

@@ -0,0 +1 @@
+