Teams that ship 90% test coverage and still page every Friday night are measuring the wrong thing. The metrics that actually predict incidents are cheap to collect, largely ignored, and have nothing to do with how many lines your test suite touches.
Why Coverage Percentage Fails as a Health Signal
Test coverage tells you which lines executed during a test run. It says nothing about whether those tests assert anything meaningful, whether they cover realistic input distributions, or whether the code under test is even load-bearing. You can achieve 90% coverage with tests that call every function and assert nothing. This happens constantly.
The more dangerous failure mode is what coverage actively hides. A codebase with 85% coverage and 400 lines of untested code concentrated in the payment processing module is far more dangerous than one with 70% coverage spread evenly across utility helpers. Coverage percentage flattens that distinction into a single number that looks fine on a dashboard and means nothing.
Drop coverage as a primary gate. Keep it as a floor check: "we don't ship if coverage on the payments module drops below 80%". Not a headline metric.
The 6 Signals Worth Instrumenting
These are the metrics that, when they move, something real has changed. Each one is collectable in a typical CI pipeline without custom infrastructure.
Cyclomatic complexity per function is the number of independent paths through a function. A function with complexity above 10 has a statistically higher defect rate. Above 15, it is almost always a bug nest. Tools like radon (Python), gocyclo (Go), and complexity-report (JavaScript) emit this in CI in seconds. The action is not to block the build on any complexity violation. Alert when a file's average or maximum complexity crosses a threshold and has moved more than 20% in a single PR.
Dependency freshness is how far your direct and transitive dependencies are from their current release. Known CVEs cluster in older versions, and the longer you let dependencies drift, the more painful each upgrade becomes. Track the number of dependencies more than 90 days behind their latest stable release. A jump of five or more in a single week is worth reviewing.
Churn-coupled complexity is where coverage metrics pretend not to go. Take every file that changed in the last 30 days and compute its cyclomatic complexity. Files with high churn and high complexity are your highest-risk surface area. This is a compound signal, not available from either metric alone, and it predicts incidents better than either one independently. OpenThunder surfaces this automatically by correlating change frequency against static analysis output on every push.
Test flakiness rate is the percentage of test runs in which a given test sometimes passes and sometimes fails without a code change. A flakiness rate above 2% on a given test suite causes engineers to start ignoring red builds, which means real failures hide in noise. Instrument this by tracking test outcomes across runs in your CI system and flagging tests that have failed at least once in the last 50 runs without a corresponding code change.
P99 build time is a proxy for how much pain you are imposing on every engineer every day. Slow builds cause engineers to batch changes, which causes larger diffs, harder reviews, and more defects per deploy. Track the 99th-percentile build time, not the mean. The mean is dragged down by warm-cache runs that do not represent the cold-start experience.
Linting suppression density is the number of # noqa, // eslint-disable, @SuppressWarnings, or equivalent inline suppressions per 1,000 lines of code. Each one is a deliberate override of a tool designed to catch problems. A rising suppression density means the team is fighting the linter instead of fixing the code. Almost always a proxy for accumulated shortcuts.
How to Collect These Without a Platform Team
You do not need observability infrastructure to start. You need a CI step that writes JSON and a place to store it.
# .github/workflows/code-health.yml
jobs:
code-health:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Compute complexity metrics
run: |
pip install radon
radon cc src/ --json --min B > complexity.json
- name: Count lint suppressions
run: |
grep -rn 'noqa\|pylint: disable\|type: ignore' src/ \
| wc -l > suppression_count.txt
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: code-health
path: |
complexity.json
suppression_count.txt
Once you have the artifacts, pipe them to any time-series store you already have. Datadog, Grafana backed by Postgres, even a Google Sheet via a cURL call to the Sheets API. The storage format matters less than the consistency. The goal in the first week is a 30-day trend line for each metric.
If you want to skip the glue work, OpenThunder ingests these signals directly and runs additional behavioral checks that static output alone cannot catch.
Establishing Baselines Before You Set Thresholds
This is where most teams skip a step and regret it. They instrument a metric on Monday, read that complexity above 15 is bad, and immediately add a CI gate that blocks merges. Then they discover their legacy payment module has had complexity of 40 for three years and now the entire team is blocked.
Run every metric for two full sprints before you set a single alert. Collect the data, compute the 90th percentile of each signal across your entire codebase, and use that as your initial threshold. Then alert on movement away from baseline, not on absolute values. A function with complexity 20 that has sat untouched for 18 months is not your problem. A function that went from 8 to 22 in a single PR is.
This distinction, between absolute thresholds and delta-based alerting, is what separates useful signals from metric theater.
Churn-Complexity Correlation in Practice
Here is what this looks like when you actually run it. Pull your git log for the last 60 days and count commits per file. Then join that against your complexity output.
import subprocess
import json
from collections import Counter
def get_churn(days=60):
result = subprocess.run(
["git", "log", f"--since={days} days ago", "--name-only", "--format="],
capture_output=True, text=True
)
files = [line.strip() for line in result.stdout.splitlines() if line.strip()]
return Counter(files)
def load_complexity(path="complexity.json"):
with open(path) as f:
raw = json.load(f)
return {
filename: max(block["complexity"] for block in blocks)
for filename, blocks in raw.items()
}
def risk_surface(churn_threshold=5, complexity_threshold=10):
churn = get_churn()
complexity = load_complexity()
hot = {
f: {"churn": churn[f], "complexity": complexity[f]}
for f in churn
if churn[f] >= churn_threshold
and f in complexity
and complexity[f] >= complexity_threshold
}
return sorted(hot.items(), key=lambda x: x[1]["churn"] * x[1]["complexity"], reverse=True)
if __name__ == "__main__":
for filename, stats in risk_surface()[:10]:
print(f"{filename}: churn={stats['churn']}, complexity={stats['complexity']}")
Run this weekly and post the top 10 results to your team Slack channel. No dashboard required. Engineers will start recognizing names on that list and self-organizing refactors around them. This alone has moved the needle on incident rate at multiple companies.
What to Do With the Signals Once You Have Them
Metrics that live in a dashboard and never change behavior are also vanity metrics. The only useful question: what decision does this metric inform?
For complexity and churn-complexity, make it visible in code review. When a PR touches a file in the top decile of your churn-complexity score, require a second reviewer. Targeted intervention, not a blanket process tax.
For flakiness rate: quarantine tests above the threshold into a separate suite that runs on a schedule but does not block deploys. Assign flaky test ownership explicitly. Unowned flaky tests never get fixed.
For dependency freshness: automate the PRs with Dependabot or Renovate, but batch minor updates into weekly PRs rather than daily noise. Reserve immediate alerts for CVE-tagged updates only.
For suppression density: add it to your PR template as a soft prompt, not a hard gate. "This PR adds 3 lint suppressions. Is there a cleaner fix?" That single question, asked consistently, changes behavior over a quarter.
Connecting Code Health to Incident Postmortems
The strongest argument for tracking these metrics is retrospective. Run a postmortem on your last five production incidents and look at where the root-cause code lived in your codebase. In my experience, across multiple production systems, the incident-causing code is nearly always in the top quartile of churn-complexity score. It is rarely in the low-coverage zone, because low coverage is evenly distributed.
Start doing this systematically. After every incident, answer three questions: what was the churn count for the file in the last 60 days, what was its cyclomatic complexity at the time of the failure, and how many lint suppressions did it contain. Track those answers in your postmortem template. Within three months you will have internal data that is more persuasive than any industry benchmark.
That data is also what you bring to engineering leadership when you need to justify time for refactoring work. "We have seven files in the top decile of churn-complexity and four of our last five incidents originated there" is a fundable argument. "Our test coverage dropped to 82%" is not.
Operationalizing the Feedback Loop
The setup that works: weekly automated report on the top 10 churn-complexity files, flakiness counts per suite, and suppression density trend. Monthly review of dependency freshness. Per-PR complexity delta reported as a comment, not a gate, for the first 90 days while you calibrate thresholds.
Do not try to fix everything at once. Pick one metric, instrument it, run it for a sprint, make one decision based on it, and see if that decision was right. Then add the next metric.
The teams that instrument six signals in week one and act on none of them have done nothing. The teams that instrument one signal and track it religiously for a quarter build the intuition to use the rest.
Put a Verification Gate on Your Pipeline
If you want the churn-complexity correlation, flakiness tracking, and suppression density signals running without building the glue yourself, that is exactly what OpenThunder was built to do. OpenThunder runs static, dynamic, and behavioral checks on every change and turns failures into fixable findings. Try it here.
The metric you track is the metric you optimize for, so track the ones that predict failures, not the ones that make the dashboard look clean.