Doc-foundry – AI writes Security detection rules, then proves they work

Detection-as-Code Foundry An adversarial multi-agent detection-engineering workflow — LangGraph · Pydantic · FastMCP · Claude Turn a threat (a CVE, a CISA KEV entry, an incident writeup, an ATT&CK technique) into a validated first-version detection — a proven starting point a detection team hardens further, not a claim of finished, forever-bulletproof coverage — plus three artifacts that ship together : a Sigma detection rule that has survived a Blue-vs-Red adversarial loop, a threat report with the round-by-round convergence history , and a proof bundle — a positive corpus the rule fires on, a benign baseline it stays silent on, an adversarial mutation set, and a computed coverage label ( real-coverage / brittle / broken ) with FP/FN numbers and a reproducibility hash. The thing nobody else produces is #3. A model can write you a Sigma rule; it cannot hand you a proof that the rule works, an adversary that provably evades the weak version, and an honest record of how the rule was hardened until it held. A bounded, fixed adversary. Red commits to two real, documented attacks in round one and re-throws those same two against the hardened rule each round — the same "built once, can't move its goalposts" discipline as the fixed test corpus. real-coverage therefore means something concrete and honest: the first-version rule fires on every attack in the corpus, stays silent on benign look-alikes, and caught both of Red's real attacks. It is a validated baseline, not a promise of coverage against every future evasion — that is the detection team's job to extend. Agents propose; the engine proves. Four role-specialized LLM agents (Threat Analyst, Blue Detection Engineer, Telemetry, Red-Team Adversary) do the creative work. Everything that decides truth — FP/FN evaluation, adversarial evasion, the coverage verdict, red-team verification, and the deployment gate — is a deterministic referee. Same input → same verdict → same reproducibility hash. Two pipelines share this agent cast. The one described above proves — Blue authors, Red attacks, a deterministic referee decides. The Review Chain reviews — draft → Red Team → revision → SOC Lead → final rule, with every note that shaped it. Different question, different deliverable; pick per what you need. 📄 See real output first: examples/joomla-icagenda-cve-2026-48939/review-history.md — a live run on a CISA KEV entry, three rule versions, eight reviewer findings. Run it pip install -r requirements.txt # or just run --evals on stdlib alone # LIVE — any threat text; a real model authors it (needs GEMINI_API_KEY or ANTHROPIC_API_KEY) python src/run.py " CVE-2026-45659 SharePoint deserialization RCE " python src/run.py " T1218.011 rundll32 remote scriptlet execution via mshtml " # CANNED — a fixture scenario, fully offline, no key (what the test suite drives) python src/run.py --threat fixtures/certutil_t1105.json # brittle→repair→proven python src/run.py --corpus atomic # positives from REAL Atomic Red Team commands python src/run.py --max-rounds 3 # self-repair loop cap (default 3) python src/run.py --evals # deterministic engine + proof + workflow tests python src/run.py --approve alice # human approves → gated deploy succeeds python src/run.py --claude " ... " # force claude-opus-4-8 authoring (ANTHROPIC_API_KEY) python src/run.py --gemini " ... " # force Gemini authoring (GEMINI_API_KEY) There is no default scenario — the Foundry takes the threat you give it. A bare python src/run.py has nothing to work on and says so. Bundles are written to Outputs// ( rule.yml , report.md , proof.json ). The Review Chain — a second, different pipeline ( --chain , and the default UI) The pipeline above asks "is this rule provably correct?" and answers with a deterministic verdict. The Review Chain asks a different question — "what would each stakeholder say about this rule, and what does it look like after their feedback is applied?" — and answers with a final rule plus the complete trail that shaped it . ingest ─▶ extract ─▶ telemetry ─▶ profile ─▶ Blue: DRAFT v1 ─▶ RED TEAM review (findings: where is this evadable / blind) ─▶ Blue: REVISION v2 (each edit cites the finding id it answers) ─▶ SOC LEAD review (findings: volume, triage, severity, tuning) ─▶ Blue: FINAL v3 ─▶ package python src/run.py --chain " CVE-2026-48939 Joomla iCagenda arbitrary file upload RCE " python -m uvicorn webui.server:app --port 8000 # → http://127.0.0.1:8000 (chain UI) # → /classic for the proof loop above Red and SOC are not the same reviewer in two hats — they pull in opposite directions. Hardening against evasion broadens a rule; broadening raises alert volume. Running both, in that order, is the point: the final rule is the one that survived both pressures, and the trail shows exactly what each pass cost. Output lands in Outputs// as final-rule.yml , review-history.md (the full progression, rendered) and review.json . A worked example — read the actual output → examples/joomla-icagenda-cve-2026-48939/ — a real run on CVE-2026-48939 (Joomla iCagenda upload-to-RCE, CISA KEV), authored by claude-opus-4-8 . Three rule versions, eight findings, the whole trail: review-history.md . Blue's first draft fired on a web daemon spawning a shell that runs recon commands . Both reviewers took it apart — in opposite directions: Red Team: "No — I'd just skip the recon commands (or use a container's busybox shell) and this rule never fires despite a full RCE." R1 mandatory recon_intent turns a lineage detection into a keyword detection · R2 on Alpine /bin/sh is busybox, so every shell clause misses · R3 literal contains dies to shell obfuscation · R4 setsid /double-fork re-parents to PID 1 and lineage fails SOC Lead: "No — this fires on your own benign cron example on day one; the parent scoping is broken and it'll bury the rota in www-data noise." S1 the OR made web-server parentage optional — any www-data process spawning a shell fired · S2 php-fpm→php is normal web-app behaviour, not RCE · S3 the alert carried one field, so the analyst pivots consoles · S4 level: high on that noise floor burns out the on-call Red pushed Blue to broaden ; the SOC Lead pushed it to narrow straight back. That tension is the product: hardening against evasion raises alert volume, and the final rule is the one that survived both pressures. SOC reviews last, so what ships is tuned for the queue — v3 made parent lineage mandatory, excluded the known-good interpreter chains, expanded the triage fields, dropped to level: medium , and carries a three-step response playbook with a containment threshold. Reviewed, not validated — and the artifact says so. This chain does not invoke the proof core. It never executes the rule against telemetry, counts no false positives, and issues no coverage verdict. A rule that has been through four professional reviewers looks finished, which is precisely why the packaged output leads with that sentence. Run the classic pipeline (or --evals ) when you need proof rather than review. The demo UI ( webui/ ) — live authoring, watch the agents work python -m uvicorn webui.server:app --port 8000 # → http://127.0.0.1:8000 ; type ANY threat, watch the loop run python tools/warm_cache.py " the exact threat you'll demo " # optional — see below FastAPI + SSE over the real graph. Two streams are interleaved and neither is scripted: node events fire when a graph node genuinely completes, and token events stream each agent's raw model output as it is generated , via the token sink on providers.py . The graph runs on a worker thread precisely because tokens arrive mid-node; the queue between them is the only coupling, and it is one-way. You watch: the LangGraph sh