Measured, not claimed

How accurate is OpenThunder?

OpenThunder’s checks run without an AI model: they read your code and your change and report what they find. For each family of checks, this page says what we ran it on, how many false alarms it raised before and after we tuned it, which real problems it found, and what it still misses.

Every number comes from a measurement recorded in an OpenThunder pull request on October 29 or 30, 2026 (cited as #number; that repository is private, so the numbers are copied here), or from a command you can run yourself. Open-source projects are named. Private production apps we have access to are not named: they are described as “11 private production apps” and similar, without names, paths, or code.

Summary
Security
Structure and health
Installed API
Tests weakened
AI behavior
Agent setup
Agent memory
Likely mistakes
Reproduce
Not measured

How to read these numbers

False alarm: an alert that a person read in the code and found was not a real problem.
Before and after: the same repositories and the same commands, run on the build before a calibration change and on the build after it. Calibration fixed the rule itself, with a test that keeps the real case alerting. There are no per-repository ignore lists.
Real issue: a finding a person checked in the code and confirmed.
The main weakness: most “after” numbers are on the same repositories the rules were tuned on. They show the noise we removed, not how the rules do on code they have never seen. Where we also ran on outside repositories, the section says so.

Summary

One line per family. The sections below give the details, the sources, and the limits.

Check family	Measured on	False alarms, before → after	Real issues found
Security	11 private production apps, a larger private web app, OpenThunder’s own code	Security gate failures on the 11 apps: 2, both false → 2, both real	Reflected cross-site scripting, a placeholder signing key, TLS verification off, unauthenticated write endpoints
Structure and health	The same 13 codebases	“Confirmed” broken client calls: 18 → 0 and 2 → 0	A 36,605-line file that was skipped is now scanned
Installed API	About 30,000 references in TypeScript, Python, Go, and Rust; 7 Java and C# repositories	Go and Rust: 29 → 0; every other sweep: 0 after	Planted mistakes only: 5 of 5, 3 of 3, 18 of 18
Tests weakened	The last 800 and 1,000 commits of two private codebases, plus 114 merge commits	Blocked: 7 commits and 27 findings → 2 blocks, both real	2 real weakenings (an exact assertion loosened; lowered thresholds)
AI behavior	404 past commits of three private codebases; a 40-case demo	10 kinds of false match removed; no count before	Model swaps, token limit changes, and prompt rewrites with no evaluation
Agent setup	31 agent setup files in 13 private repositories	0 after (no before count)	None in these repositories; detection shown on fixtures only
Agent memory	42 private repositories; about 6,000 changed files in past commits	Code-comment rule: 2 false blocks in live use → 0 in a 300-commit replay	1 instruction that pointed to a renamed file
Likely mistakes	25 open-source repositories plus private ones, in seven languages	Go and Rust: 95 → 8 alerts (1 false)	13 confirmed real on open-source code (Java, C#, Go, Rust)

Security

The security gate stopped failing on false alarms and started failing on real problems in the same apps.

The scanner, the Security Lens, and the openthunder security check gate: injection, secrets, cross-site scripting, missing authentication, TLS settings, and more. Two people-run security reviews of the 11 apps listed the issues to find; the rules were then fixed until the noise went away and the real issues showed.

Measured on	11 private production apps (TypeScript, JavaScript, and Python; serverless, Express, Fastify, FastAPI, Next.js), a larger private web app (about 3,400 source files), and OpenThunder’s own code. Read-only copies; nothing was written into any repository. #448, #451
False alarms, before → after	Security gate on the 11 apps: 2 failed, both on false alarms (design tokens read as a secret in a log; a default key the app already refuses in production) → 2 fail, both on real issues. #451 OpenThunder’s own gate: failed on 2 false conditions (a documented TLS opt-out; 16 planted test fixtures) → passes. #448 31 false “route without an auth guard” on one FastAPI app → 0. #451 SQL injection on queries with bound parameters, larger app: 4 of 4 were false → 0. Three “proven” server-side request forgery findings where only a header carried request data → 0. Nine `yaml.load` alerts in JavaScript, where js-yaml has not run code on load since version 4 → 0. #448 Security Lens issue titles: 21 → 16 on OpenThunder, 20 → 15 on the larger app. #448
Real issues it found	A reflected cross-site scripting hole on a public page of a Python app, and a placeholder signing key used for every login token in the same app. Both now fail the gate. #451 TLS certificate checks turned off on a database connection. The scanner knew; the gate now agrees. #451 Three public endpoints that write to a database with no authentication: exactly the three the review named. #451 Request data put into HTML e-mail without escaping, in two apps’ contact forms. #451 The larger app’s 7 real high findings (6 default secrets and one SQL injection) stayed exactly as they were. #448 A miss fixed: a hardcoded GitHub token in a change was not Blocked; now it is, naming the file and line, never the value. #448
Compared with the people-run reviews	Of the review items, 18 are now reported (some only as low or “worth a look”, one only partly), 8 are not, and 1 is silent on purpose (the review rated it low and the code returns 403). This is not independent recall: the review list and the rule fixes were made together. #451
Known limits	Not found: logic flaws with no fixed code shape (a login token not bound to its tenant, a lockout counted per process, a free-text role field), stored cross-site scripting that crosses functions, and a missing request-size limit. #451 The Security Lens reads at most 5,000 files and does not say so yet; paths containing `demo/` are treated as non-production. #451 Rust: path traversal, server-side request forgery, and weak hashing are not checked; the scan says so. #475

Structure and health

Fewer false “broken call” and “no tests” answers on real apps; the real large files and complex code still show.

The architecture map (routes, client calls, database writes), test detection, repository health, churn hotspots, and code health, plus the Check’s answer on a change.

Measured on	The same 13 codebases as Security (3 of the 11 apps are not git repositories, so only 8 have a Check on a commit). #448, #452
False alarms, before → after	“Confirmed high” broken client calls: 18 → 0 on the larger app (every one hit a router mounted with middleware) and 2 → 0 on OpenThunder (benchmark fixtures). #448 Broken-call findings on the 11 apps: 9 → 0. Routes that nothing in the repository calls: 447 → 266. These are informational, and the remaining ones were not each reviewed; in one app, the 2 left are a payment webhook and an endpoint called from outside. sums of the #452 table Test framework found: 0 of the 11 apps → 7 of 11 (pytest, Vitest, Playwright, `node:test`), plus the larger app (Jest and Vitest). The other 4 still show none. #452 Check on each app’s last commit: Needs review on 7 of 8 → 3 of 8. #452 “The changed code has no tests” on a file that 9 test files import → tests found by import. #448 A real change that only edited text and punctuation was Blocked by 30 “no test covers it” items: 51 of its 54 changed code files now read as text-only, and the 3 real code changes still count. #452 The larger app’s repository health listed “no test command” and “no CI” as high risks; it has both (5 risks → 0). #448
Real issues it still shows	A hand-written 36,605-line file that was silently skipped is now scanned and reported as large and complex; real oversized modules, complexity, duplication, and empty catch blocks are unchanged; security findings did not change on any repository. #452
Known limits	Routers built in another file are not prefixed (Go, Rust); Vercel rewrite conditions are not read; words like “if” in page text can count as decision points. #452, #475

Installed API

On every line of the code we swept, zero false “this API does not exist” blocks after tuning.

The Check blocks a change that calls a function, option, or import that does not exist in the version of a dependency you actually have installed, and suggests the closest real name. It reads the installed packages; it does not ask a model.

Sweep (every line, not only a change)	Files	References checked	False blocks after	Source
OpenThunder (TypeScript)	1,687	5,280	0	#458
Larger private web app (TypeScript)	3,363	15,841	0	#458
3 of the private apps (TypeScript)	418	4,142	0	#458
A private Python backend, with its virtual environment	57	363	0	#458
Go: gin RealWorld, fzf, caddy	not counted	4,051	0 (with Rust: 29 before)	#475
Rust: ripgrep, mdBook, an axum and sqlx RealWorld app, two private desktop shells	not counted	877	0	#475
Java and C#: 7 real repositories	not counted	not counted	0	#474

Planted mistakes caught	TypeScript: 5 of 5 (a misspelled import, an invented option key, two invented methods, a method that does not exist in the installed client). Python: 3 of 3 (a misspelled import, a keyword argument the installed version does not have, a misspelled function). Java and C#: 18 of 18. #458, #474
Real issues it found	None in the sweeps: no unplanted wrong-API call turned up, so detection is shown only on planted mistakes. In TypeScript projects most of these are also compiler errors, and the Check says so; the new coverage is plain JavaScript, Python, and agents that never run the compiler. #458
Known limits	Go and Rust check names only (not methods or fields); Java has no instance-member checks; C# does not check `Type.Member`; Python follows instances only within one file; a package that is not installed is Not checked, never passed. #458, #474, #475

Tests weakened

Replayed over 1,800 recent commits and 114 merge commits of two codebases, it blocked twice, and both were real.

Did this change make the tests prove less? Removed or loosened assertions, skips, deleted tests, production code that special-cases tests, and test settings that run less.

History replayed	Changes	Flagged	Blocked
OpenThunder, last 800 commits	800	3 (Needs review)	0
OpenThunder, merge commits	54	2	1, real: an exact assertion loosened to “is truthy” while the code it tests changed
Larger private web app, last 1,000 commits	1,000	10	1, real: two minimum thresholds lowered while the code changed
Larger private web app, merge commits	60	1 (Needs review)	0

Before tuning	7 OpenThunder commits and 27 findings on the larger app were Blocked. #464
Not measured	The flagged Needs review items were not each labeled real or false in the record. The replays covered TypeScript and JavaScript history only; Python, Java, C#, Go, and Rust are covered by fixtures, not replays. #464, #473 to #475
Known limits	It reads text, not a parse tree; it cannot tell why a test changed (a deliberate behavior change is reported, and a person accepts it); new test files are never findings. #464

AI behavior

Prompt, model, and setting changes are caught in real history; how many of those flags were right is not yet counted.

A change edits a prompt, model, or setting with no evaluation

Measured on	Past commits, each checked against its parent: a private app that grades answers with a model (last 200 commits: 27 flagged), OpenThunder (last 200: 14 flagged), a small private app (all 4: 0). No model was called. #460
Real changes it found	A scoring model swapped for a different one, a grader’s token limit raised from 4,096 to 12,000, and a rubric prompt rewritten, none with an evaluation. In the 40-case demo, a model swap that drops the pass rate from 95% to 50% is Blocked, and a reworded prompt that holds 95% is not. #460, `examples/ai-eval-demo`
False alarms	10 kinds of false match were found in the replays and removed, each with a test (for example, `font-weight` read as a grading weight, a model picker key read as a model id, a graph seed read as a sampling setting). Two misses were fixed. There is no before count. #460
Not measured	The 27 and 14 flags were not each graded real or false in the record; examples were checked. #460
Known limits	Pattern based, not data flow: a prompt built from many helper strings, or more than one import away from the model call, can be missed. #460

AI features in your app (mapped to the OWASP Top 10 for Agentic Applications)

Measured on	The larger private app (91 AI files), OpenThunder (21 AI files), and three private apps with little or no AI code. #459
Results	Larger app: 10 notes that large fixed system prompts are sent with no prompt caching, checked by hand and true. 4 false positives were removed (an admin secret compared to a header is not an agent’s identity). OpenThunder: no AI tracing (a note) and 57 “worth a look” items, not each reviewed. Apps with no AI surface show Not checked everywhere. #459
Known limits	Guards and user scoping are recognized by common SDK shapes and names; a guard with another name is missed, and a well-named guard with no real check is trusted. Three of the ten OWASP risks need a person. #459

Agent setup

No false alarms on real agent setups; we have no real setup problem to show it catching yet.

What the files in a repository let a coding agent do: MCP servers, permission allowlists, hooks, skills, and instruction files.

Measured on	OpenThunder and 12 private repositories: 31 setup files. #456
Result	0 issues and 0 “worth a look”. OpenThunder’s own setup was listed correctly. #456
Detection	Shown on fixtures only: 26 rule tests that must alert and must stay silent, including three shapes from 2026 incidents (a server swapped under the same name, inline interpreter code, always-allow trust), and 7 Check tests. #456
Known limits	Codex TOML is parsed loosely; hook scripts outside the repository are Not checked; hidden Unicode and hostile comments are handled by the AI security rules instead. #456

Agent memory

One real stale instruction found in 42 repositories, and two false blocks in live use fixed.

Is what your coding agents read still true, consistent, small enough, and free of hidden orders or secrets?

Measured on	41 private repositories plus OpenThunder (full audit), then a replay of the Check over each repository’s last 150 commits: about 6,000 changed files. #457
Result	Replay: 0 Blocked, 1 Needs review (real: an instruction file pointing to a document renamed in July), 1 “worth a look” (a note in an HTML comment). Size guidance on OpenThunder itself: its instructions send an agent to read about 80,000 tokens of documents before starting. #457
False alarms, before → after	The rule for hostile orders in code comments Blocked two ordinary developer comments in one day of use. After the fix, a replay of 300 commits found: OpenThunder 1 false finding → 0; the larger app 0 → 0. #468
Known limits	Token counts are estimates (characters divided by 4); a path written in plain prose is not checked; it does not judge whether a rule is still wise. #457

Likely mistakes, per language

A likely mistake makes the Check say Needs review, never Blocked. Here is how often those alerts were right.

TypeScript, JavaScript, and Python

Measured on	6 private repositories and 3 open-source ones: full-stack-fastapi-template, an Express RealWorld app, and Flask. #473
Result	No high finding from these rules in any of them. No medium in the three open-source projects. 10 kinds of noise were removed during calibration; there is no before count. #473
Not measured	File counts, medium counts on the private repositories, and confirmed real issues are not in the record. Detection is shown on fixtures. #473
Known limits	Async functions declared in other files are not known; request input is traced within one function, not across calls. language-support.md

Java and C#

Repository	Files	Needs review alerts	Read by a person
spring-projects/spring-petclinic	50	0
google/gson	264	0	3 “worth a look” in tests and samples
javalin/javalin (Java part)	64	0
macrozheng/mall	524	1	Real: unrestricted Jackson default typing for a cache
apache/commons-lang	629	2	Real but by design: two undocumented `return null`
dotnet-architecture/eShopOnWeb	254	2	Real: an `async void` handler; a configuration value forced non-null
davidfowl/TodoApi	40	1	Real: a claim value forced non-null
App-vNext/Polly	802	0	5 “worth a look” in documentation snippets
Private repositories with Java or C#	322	2	The same as eShopOnWeb (a copy of it)

Source: #474 and docs/architecture/language-support.md. The patterns that alerted wrongly during calibration are now must-stay-silent tests; there is no before count. Limits: no parse tree or type inference, no Kotlin rules, no data flow across methods.

Go and Rust

Measurement	Before tuning	After
Needs review alerts on 13 repositories (about 3,000 files)	95	8: 7 real (dropped errors in fzf and hugo; unwraps on a file read, a config parse, and input in mdBook), 1 false (an unwrap in axum its author documents as unreachable)
`InsecureSkipVerify` in caddy	4	0 (all four were operator opt-ins)
Loop-variable capture, hugo forced to Go 1.21	8	0, and 1 real capture missed; nsq: 1, real
“Worth a look” (shown, never counted)	n/a	42, mostly deliberate

Source: #475 and docs/architecture/language-support.md. Go: gin RealWorld, fzf, chi, caddy, hugo, gin, nsq, cobra. Rust: axum, tokio, ripgrep, serde_json, mdBook, an axum and sqlx RealWorld app, and two private desktop shells (0 alerts). Limits: no type checker, so each rule is narrow by design; Rust indexing panics are not checked.

How to reproduce

Every check on this page is keyless and runs on your machine; nothing leaves it. Run the same commands on any repository, including the open-source ones named above.

openthunder security check            # the security gate
openthunder scan                      # scanner findings, by severity
openthunder api-check --all           # installed-API sweep over every line
openthunder tests-weakened --base <parent> --head <commit>   # replay one past commit
openthunder ai-readiness              # AI features in your app
openthunder eval --detect             # AI behavior changes, nothing run
openthunder agents audit              # agent setup
openthunder agents memory             # agent memory
openthunder check                     # the Check on your current change

Replay scripts	Not public yet. The replays over past commits were one-off scripts run on read-only copies. To replay a history yourself, loop over `git rev-list` and run `openthunder tests-weakened --base <parent> --head <commit>` for each commit.
Report honesty	`openthunder dev lint-assessments [repo]` runs every report OpenThunder can produce for a repository and fails on rules a machine can check: unrounded numbers, Not checked counted as a failure, style advice shown as a blocker, a ship answer for a whole repository. It checks how results are reported, not whether detection is right. Recorded runs: 0 violations on OpenThunder, eShopOnWeb (20 reports), and axum (20 reports). #473, #474, #475
Planted-issue benchmark	The Accuracy Benchmark runs the scanner on 9 small fixtures with a known planted issue or a known-clean control: 7 of 7 planted issues found, 0 of 2 clean controls flagged (generated October 26, 2026). It is small and was written by us.

What these numbers do not show

No independent or third-party evaluation. We ran these measurements on our own rules.
Most “after” numbers are on the repositories the rules were tuned on. Performance on code we have never seen is likely to be worse.
What OpenThunder misses was measured only through planted mistakes, fixtures, and two people-run security reviews of the 11 apps. Nobody has counted the real bugs in these repositories that neither a person nor OpenThunder found.
Where a section says flags were “not each graded”, we do not know their false-alarm rate.
The private-repository numbers cannot be reproduced by you. The open-source ones can.
Each number is one run on one commit of each repository, on October 29 or 30, 2026. Rules have changed since; a new run can give different numbers.
Optional AI-written explanations (with your own key) are not measured here.

If a number on this page turns out to be wrong when you check it, tell us and the page changes. Questions: support@skillstechtalk.com.