A team at a mid-size fintech ships a change on Friday afternoon, watches the CI pipeline go red on a test that has failed and re-passed seventeen times in the last month, retries it twice, it goes green, and they merge. By Monday, production has a race condition in the payment reconciliation service that the test was actually trying to catch. The retry hid the signal. This is the real cost of flaky test triage done wrong: not the wasted CI minutes, but the broken verification contract.
The Two-Class Problem Most Teams Collapse Into One
Teams almost universally treat flakiness as a single phenomenon. A test fails intermittently, so it gets retried or quarantined, and the pipeline moves on. That framing collapses two fundamentally different failure modes into one response, and that collapse is where the damage happens.
Infrastructure-origin flakiness is noise. A test fails because a shared Docker host ran out of file descriptors, because a test database container took 800ms longer to accept connections, because a DNS lookup in a network-isolated environment timed out. The test itself is fine. The application code it covers is fine. The environment lied.
Application-origin flakiness is signal. A test fails intermittently because the code under test has a real timing dependency, a shared mutable state problem, or a non-deterministic data access pattern. The test is doing exactly what it should: surfacing a genuine defect that only manifests under certain scheduling or load conditions.
Blind retries treat both classes identically. When you retry an application-origin flake and it passes, you have not confirmed correctness. You have confirmed that the defect did not manifest this time. That is a meaningfully different thing, and treating it as equivalent is how the fintech team ends up merging a race condition on a Friday.
Building a Failure-Origin Classifier Before You Build a Retry Policy
The right investment order is: classify first, respond second. Most teams do this backwards, reaching for retry configuration in their CI YAML before they have any instrumentation that tells them what they are actually retrying.
Here is a concrete classification pipeline you can build in stages.
Stage 1: Timing histograms per test. For every test in your suite, record wall-clock duration on every run and persist it. A test that occasionally takes 4x its median duration and fails is almost certainly hitting an infrastructure ceiling, not a code defect. A test that fails in roughly its normal duration is more likely expressing a genuine nondeterminism. You can build this with a simple test reporter that writes timing data to a time-series store like InfluxDB or even a Postgres table with a test_id, duration_ms, outcome, branch, and runner_id column. Once you have 30 days of data, the histogram shapes tell you almost everything.
Stage 2: Environment fingerprinting at the run level. When a test fails, capture runner metadata: CPU steal percentage, memory pressure, available file descriptors, container startup latency for any service dependencies. If failures cluster on a specific runner label, a specific time of day, or a specific resource contention pattern, that is infrastructure noise. If failures distribute uniformly across environments and time, that is application signal. Most CI platforms expose enough metadata to make this tractable without writing a custom agent.
Stage 3: Cross-branch flake correlation. A test that fails on a feature branch but not on main under identical infrastructure conditions is more likely carrying application-origin nondeterminism introduced by that branch's changes. A test that fails at the same rate on main, on every feature branch, and on scheduled nightly runs is almost certainly infrastructure noise. Correlating flake rate by branch lets you catch the Friday-afternoon merge scenario: a new intermittent failure that only started after a specific commit is not a flake to suppress, it is a defect to investigate.
What to Actually Do With the Classification
Once you have failure-origin labels, your response policy can be precise instead of blunt.
For infrastructure-origin failures: retry is appropriate, but cap it at one retry and alert on sustained retry rates above roughly 5%. If a test is retrying successfully more than one in twenty runs, your environment has a reliability problem that retries are papering over. That deserves its own remediation track, not normalization.
For application-origin failures: quarantine, never retry. Move the test to a non-blocking suite, file a defect with the full failure trace, and set a two-week SLA for investigation. The quarantine is not a permanent home. It is a staging area for defects you have acknowledged but not yet fixed. A quarantine list that grows without bound is a signal that your team's defect SLA is broken, not that the tests are wrong.
This is the position worth defending: a quarantined test that is not fixed within its SLA should be re-evaluated for promotion back to blocking, not deleted. Deleting it removes coverage. Keeping it non-blocking indefinitely trains engineers to ignore quarantine alerts. Neither outcome is acceptable.
Instrumentation That Actually Surfaces the Classification
The tooling here does not need to be exotic. What it needs to be is persistent and queryable.
A failure-classification table with columns for test_id, failure_class (infrastructure or application), first_seen_commit, last_seen_commit, flake_rate_30d, and current_status (blocking, quarantined, fixed) gives you everything you need to run a weekly triage rotation. The rotation is not a meeting where people guess. It is a query against real data: show me all application-origin flakes quarantined more than fourteen days ago, ordered by flake rate.
OpenThunder's behavioral check layer integrates directly with this kind of failure taxonomy. Rather than treating every failing check as equivalent, it surfaces failure context alongside the finding, which makes the origin classification step substantially less manual on pipelines with high test volume.
Timing histogram divergence thresholds worth starting with: flag a test for infrastructure review if its p95 duration exceeds 3x its p50 in the last 500 runs. Flag a test for application review if it fails at least 3 times in 100 runs with duration inside 1.5x its p50. Those numbers are not magic. They are starting points you tune based on your suite's observed distributions after the first 30 days of data.
The Verification Gate Contract and Why Blind Retries Break It
A CI verification gate makes an implicit promise: green means the code is correct enough to merge. Every time you add a retry to paper over a failure you do not understand, you are weakening that promise in a way that is invisible to everyone reading the green checkmark.
This is the argument that usually gets pushback. Engineers say: our pipeline would be perpetually red without retries. That is true, and it is also the correct signal. A pipeline that is perpetually red without retries is telling you that your test infrastructure has a reliability problem, or that your application has genuine nondeterminism. Both of those are real problems. Retries do not fix them. They hide them behind a green status that everyone learns to trust a little less each time.
Track your retry rate as a first-class pipeline metric alongside build duration and failure rate. If your retry rate trends up over 90 days, your verification gate is degrading. OpenThunder exposes this kind of gate health trend as a native metric, which means you can alert on it rather than discovering it post-incident.
Teams that invest in failure-origin classification instead of just retry configuration find something counterintuitive: their effective flake rate drops faster. Not because they suppressed more tests, but because classifying failures correctly creates accountability. Infrastructure failures get routed to platform engineers. Application failures get routed to the team that owns the code. Ownership produces fixes. Quarantine produces learned helplessness.
The single most damaging thing a platform team can do is give developers a one-click retry button without any classification tooling underneath it. You are not reducing friction. You are selling the verification contract for a few seconds of convenience.
Put a verification gate on your pipeline
If you are operating a CI pipeline at any meaningful scale, the classification infrastructure described here is not optional tooling. It is the foundation that makes your gate meaningful. OpenThunder runs static, dynamic, and behavioral checks on every change and turns failures into fixable findings with origin context already attached. Try it here.
A verification gate that cannot distinguish noise from signal is not a gate: it is a rubber stamp with extra latency.