A single flaky test in a high-frequency CI pipeline can generate more than 10,000 false-positive failures per year without anyone filing a single bug about it. That number sounds absurd until you realize the reason nobody files a bug is precisely the problem: engineers have already learned to ignore it.
The Alarm Clock Problem
There is a well-documented psychological phenomenon where people who live near a train track stop hearing it within weeks. CI pipelines work the same way. When your test suite produces a steady background hum of non-deterministic failures, engineers recalibrate their signal threshold upward. They start merging on red. They start adding --retry 3 flags to hide the noise. They stop reading failure output because they have learned, through painful experience, that most failures are not their fault.
This is worse than having no tests at all. A broken alarm clock that goes off randomly does more damage than no alarm clock, because it trains you to sleep through the real fire.
Flaky test blindness is the organizational state where engineers have lost the ability to distinguish a real regression from a race condition in a 3-year-old integration test. Once you reach that state, your test suite is a liability, not an asset.
The fix is not "just fix the flaky tests." That is the naive answer, and it does not survive contact with a 4,000-test monorepo. The fix is building the detection infrastructure that tells you which tests are actually flaky, how flaky they are, and whether a given failure is signal or noise.
What "Flaky" Actually Means Statistically
Most teams define a flaky test informally: it fails sometimes without a code change. True, but not useful operationally. You need a quantitative model to prioritize remediation and to build a reliable pass/fail gate.
The right mental model is flakiness rate: the probability that a given test produces a non-deterministic failure on any given run, independent of the code under test. Estimating this requires repeated execution under controlled conditions.
A simple approach is tracking consecutive outcomes per test over a rolling window. If test T ran 100 times in the last 7 days and failed 12 times on commits that were otherwise green, its estimated flakiness rate is roughly 12%. But this naive estimate has two problems:
- Attribution: A failure on a PR that also introduced a real bug is ambiguous. You cannot cleanly separate the flakiness rate from the regression rate without a control group.
- Survivorship bias: Long-running tests that flake 40% of the time get disabled or skipped. What remains in your suite is a biased sample of moderate flakers.
The cleanest approach is running a flakiness detection job that executes each test in isolation on a fixed known-good commit, multiple times in parallel, and records the outcome distribution.
import subprocess
import concurrent.futures
from collections import Counter
def probe_test_flakiness(test_id: str, commit: str, runs: int = 30) -> float:
"""
Runs a single test N times against a fixed commit.
Returns estimated flakiness rate (0.0 to 1.0).
"""
def run_once(_):
result = subprocess.run(
["pytest", test_id, "--tb=no", "-q"],
capture_output=True,
env={"GIT_COMMIT": commit}
)
return result.returncode
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
outcomes = list(pool.map(run_once, range(runs)))
counts = Counter(outcomes)
failures = counts.get(1, 0)
return failures / runs
This is the kernel of every serious flakiness detection system. The fixed commit is the control. Parallelism keeps the job fast enough to run on a schedule. The output is a number, not a label, which means you can set thresholds rather than making binary calls.
A test with a flakiness rate above 5% should be quarantined immediately. Between 1% and 5%, it belongs in a remediation backlog with a P2 priority. Below 1%, it is noise in your estimate that is not worth acting on yet. Those thresholds are defensible but not universal: a pipeline that runs 500 tests on every commit will see different aggregate false-positive rates than one that runs 50.
Building the Quarantine Layer
Detection without action is just a metrics dashboard nobody looks at. The operational lever is test quarantine: a mechanism that runs flaky tests on every CI run but does not let them block merge.
This sounds obvious. The implementation details are where teams make mistakes.
The naive quarantine approach is maintaining a static skip list. You add the flaky test to a file, it gets excluded from the blocking gate, and someone adds a TODO to fix it later. The TODO never gets resolved. Six months later you have 200 tests in the skip list and no idea whether any of them still flake.
The better approach treats quarantine as a temporary state with a TTL. A quarantined test is still executed on every run, its results are recorded, and after N consecutive passes (say, 20) it is automatically promoted back to the blocking tier. If it fails during the quarantine period, the failure is logged and routed to the owning team, but it does not block the pipeline.
This creates a self-healing loop instead of an ever-growing skip list. The key implementation detail is ownership metadata: every quarantined test needs an attributed owner who receives the failure notification. Without that, the notification goes nowhere and the loop breaks.
The ownership model also forces a useful organizational conversation. When engineers see their name on a quarantine notification, they fix the test. When the notification goes to a generic #ci-alerts channel, it gets ignored. This is not a technical problem. It is an incentive design problem.
OpenThunder's detection layer surfaces per-test flakiness scores and routes failures to their origin commit automatically, which short-circuits the attribution problem without requiring manual ownership tagging.
The Correlation Trap in Flakiness Detection
Here is the non-obvious failure mode that catches even experienced platform teams: flakiness rate is not independent across tests.
Tests that share infrastructure, like a Postgres test database, a Redis instance, or a mock HTTP server, have correlated failure modes. When the shared resource gets into a bad state, multiple tests fail together. If you measure each test's flakiness rate independently, you underestimate the blast radius of the underlying cause and over-count the number of distinct problems to fix.
The diagnostic move is clustering tests by co-failure patterns. If tests A, B, and C fail together 80% of the time when they fail at all, the root cause is almost certainly in their shared setup or teardown, not in the tests themselves. Treating them as three independent flaky tests results in three wasted investigations.
A cosine similarity calculation on the binary failure vectors across runs will surface these clusters. You do not need anything fancier than that. The output tells you where to point a single fix instead of three.
This same pattern generalizes. If flakiness spikes during certain time windows, you are probably looking at a resource contention problem related to job concurrency, not a test logic problem. If flakiness spikes on specific runner types, you are looking at a machine configuration issue. Temporal and environmental correlation is data. Use it before you start rewriting tests.
For teams wiring this kind of analysis into their pipeline, OpenThunder runs behavioral checks on test suites that include this correlation pass, so you are not discovering the infrastructure root cause three rewrites too late.
Turning Detection Into a Durable Gate
The end state you are building toward is a CI gate where a failure means something. That requires three properties working together:
- Precision: The gate does not fire on known flaky tests.
- Recall: The gate does fire on real regressions, including new flakes introduced by the current change.
- Latency: The gate produces a result fast enough that engineers do not work around it.
Precision and recall are in tension. The more aggressively you quarantine, the higher your precision but the worse your recall on new flakes. The right balance depends on your release cadence. A team shipping multiple times per day needs higher precision to maintain merge velocity. A team shipping weekly can afford more conservative quarantine thresholds.
Latency is often the constraint that blows up the other two. A flakiness probe that takes 45 minutes produces results nobody waits for. The target for a flakiness detection job is under 10 minutes for the per-test probing, which means parallelization is not optional.
The structural recommendation: run flakiness probes on a schedule (every 6 to 24 hours, depending on how often your test suite changes), publish the results to a per-test metadata store, and have your CI gate consult that store at the start of each run to determine which tests are in the blocking tier. This decouples detection latency from merge latency entirely.
New tests default to the blocking tier for their first 20 runs before a flakiness rate is established. This is conservative, but the cost of blocking a merge on a new flaky test once is far less than the cost of shipping a regression because a new flaky test was pre-emptively quarantined.
Put a verification gate on your pipeline
If you have read this far, you know the problem is not just writing less flaky tests. It is building the detection, attribution, and quarantine infrastructure so that failures mean something again. OpenThunder runs static, dynamic, and behavioral checks on every change and turns failures into fixable findings, including flakiness correlation analysis that tells you whether the root cause is in the test or in the infrastructure it runs on. Try it here.
A test suite that engineers ignore is not a safety net; it is a false sense of one.