security-benchmark

Open-source, 100% reproducible AI Agent Runtime Security Benchmark & Sandbox Environment (RFC-010 Draft Protocol).

Open, privacy-bounded assurance for AI agents: containment provenance, identity passports, authorization twins, OTel evidence, CI gates, and OSCAL.

Adversarial security benchmark for agent authorization: does a compromised agent's policy-violating proposal become an unauthorized external effect? 73 trials, nine families, an independent oracle, per-mechanism ablation, confidence intervals. 0 unauthorized effects in 61 attack trials (95% CI [0.0%, 5.9%]). Reproduction is partial.

Open deterministic security tests for unsafe multi-agent handoffs and authority escalation.

Open-source benchmark for adversarial evidence attacks on LLM-based cybersecurity auditors, targeting ACM AsiaCCS 2027.

Local-first workbench to run, inspect, compare, report, and gate OpenAI Codex Security scans.

Deterministic security benchmark for tool-using AI agents

Vendor-neutral benchmark measuring how MCP security proxies/gateways DEFEND against 22+ attack vectors — crosswalked to NIST AI RMF & OWASP LLM/Agentic Top 10. CI-gated, reproducible, DOI-cited. Submit your tool to the leaderboard.

FreightSkillBench is a reproducible benchmark for evaluating document-to-transaction integrity, prompt-injection risk, and security controls in AI-enabled shipping and logistics workflows.

GitHub action for Maester

Production-grade microservices security benchmark featuring OWASP Top 10 logic exploits, automated remediation, custom Semgrep SAST rules, and CI/CD DevSecOps gates.

Commit-pinned cal.diy source corpus; Vybscan ground-truth oracle available in benchmark-results

The core repository for the Maester module with helper cmdlets that will be called from the Pester tests.

Internal PyPI SCA precision and recall benchmark corpus

Automated adversarial security testing for AI agents. Deploys an LLM-powered attacker against tool-using systems, validates violations via deterministic oracles, and produces reproducible vulnerability reports with causal attack graphs.

Open AI-for-security validation benchmark: non-LLM scorer + a SOTA-validation loop. Labeled positive corpus withheld pending coordinated disclosure.

Reproducible benchmark for smart-contract security tools, measuring precision, recall, and false positives against executable PoCs and versioned ground truth.

Product-security LLM benchmark harness for realistic AppSec, supply-chain, and LLM application security evaluations.

Internal Go Modules SCA precision and recall benchmark corpus

The public adversarial evaluation suite for Signetry: measures attack success rate (ASR) and utility-under-defense for coding-agent prompt injection, skill/MCP poisoning, and memory-injection threats, governed by signetry-core.