Accuracy Benchmark: ground truth, not grades.
Each row is a fixture with a KNOWN planted issue (or a known-clean control). OpenThunder's keyless detectors run against it, and we publish what was expected vs what was detected. Misses and false positives are shown honestly: that is what makes an accuracy claim credible.
Keyless, no AI. Generated 2026-07-27. ← Evidence Gallery
| Case (planted ground truth) | Kind | Expected | Detected | Result |
|---|---|---|---|---|
| Breaking API removal Removing a public export breaks every consumer; a line diff hides it, a contract diff catches it. |
change | flag breaking API removal | CAUTION: 1 breaking removal(s) (chargeCard) | detected |
| OS command injection User input concatenated into a shell command lets an attacker run arbitrary commands. |
scan | detect high command | high injection (SEC-command-injection) | detected |
| eval() of untrusted input Evaluating request data as code is arbitrary code execution. |
scan | detect medium eval | high injection (SEC-eval) | detected |
| Hardcoded cloud credential A committed AWS key is an immediate credential-exposure risk. |
scan | detect high secret | critical secrets (SEC-secret) | detected |
| Safe code (false-positive challenge) Parameterized query + env config + no secrets must NOT produce a high/critical finding. |
scan | no high/critical finding | no findings (clean) | detected |
| Safe change with coverage (false-positive challenge) A small refactor whose test is updated alongside it must not be hard-blocked (HOLD). |
change | verdict not HOLD | CAUTION (0 uncovered) | detected |
| SQL injection (string-built query) Request input concatenated into a SQL string is the textbook injection. |
scan | detect high sql | high injection (SEC-sql-injection) | detected |
| Unsafe AI agent (LLM output to shell) An LLM agent with side-effect tools and no guardrails; the AI-security lens should flag the risk. |
scan | detect low | low LLM10 Unbounded Consumption (AISEC-unbounded) | detected |
| Untested auth-logic change Changed auth logic with no test is unproven; the verdict should not be a clean SHIP. |
change | verdict not SHIP | HOLD (0 uncovered) | detected |
Reproduce: the fixtures live in apps/docs/benchmark/cases/; run
node apps/docs/scripts/gen-benchmark.mjs. Scan cases run
openthunder scan --all; change cases git-init the fixture and run
openthunder can-i-ship. No AI, no network.
Scope: each case targets patterns and languages OpenThunder's keyless detectors cover, and the “detected” column shows exactly what fired (including severity). The fixtures are public; extend them to probe further or to hold the engine to a regression bar.