Building a Verification Pipeline for Agent-Written Pull Requests

An agent-authored PR needs a different review pipeline than a human one. The stages worth adding, and the ones that turn out to be unnecessary theater.

OpenThunder Editorial · 2026-08-18 · AI-assisted article

Adding a "GPT reviewer" bot is rarely the right first move for verifying agent-written code. A deterministic pre-merge gate that catches structural violations before any AI opinion runs is.

That inversion matters because most teams building with coding agents reach for AI-on-AI review first: have another model critique the PR, score it, maybe post a comment. It feels symmetrical. It is mostly theater. An LLM reviewer will confidently approve code that breaks your type system, violates your migration policy, or introduces a subtle N+1 query, because those failures live in context the reviewer model does not hold. The checks that actually catch regressions are the boring, deterministic ones you already trust for human PRs, plus a handful of agent-specific layers you almost certainly do not have yet.

This article is about building those layers in the right order.

The Fundamental Difference Between Human and Agent PRs

Human authors self-censor. They know the repo's conventions, remember the last incident, and feel social pressure from code review. Agents do none of those things. An agent will cheerfully introduce a raw os.getenv call in a codebase that has a config abstraction layer, not because it is incapable of better, but because nothing in its context window told it the rule existed.

Agent-authored PRs have a different failure distribution than human ones. Human PRs fail at logic boundaries: wrong algorithm, missed edge case, architectural misfit. Agent PRs fail at convention boundaries: correct logic, wrong abstractions, missing policy compliance, subtle security regressions from plausible-looking patterns.

Your verification pipeline needs to be shaped to that distribution, not cloned from what works for human review.

Stage 1: Deterministic Gates First, Always

Before anything else runs, your pipeline should block on the checks that have zero false-negative rate for the failure classes you care about most.

# .github/workflows/agent-pr.yml (excerpt)
jobs:
  deterministic-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Type check
        run: npx tsc --noEmit
      - name: Lint (policy rules only)
        run: npx eslint --rule 'no-process-env: error' src/
      - name: Dependency license scan
        run: npx license-checker --onlyAllow 'MIT;Apache-2.0;BSD-3-Clause'
      - name: Migration safety check
        run: python scripts/check_migrations.py --require-reversible

Notice the lint step runs only your policy rules, not style preferences. Style is noise in an agent PR. Policy violations are blockers. Separate them in your config so agent PRs do not get drowned in formatting complaints while a genuine no-process-env violation slips through at severity "warning".

The migration check is agent-specific and worth implementing. Agents generating schema changes rarely produce reversible migrations unless explicitly prompted. A 30-line script that greps for DROP COLUMN without a corresponding rollback step, or an ALTER TABLE without a down migration, will catch more regressions than any LLM reviewer will.

Stage 2: Behavioral Coverage Verification

This is the stage most teams skip. It is also the most important one.

When a human writes a function, they usually write at least a minimal test because the social cost of zero coverage is visible in review. Agents write tests when prompted and skip them when not. Even when they do write tests, the tests tend to be happy-path mirrors of the implementation: they pass because they test the code that was written, not the behavior that was intended.

The check you want here is not "coverage percentage." It is coverage of the specific lines and branches introduced by this PR.

# Differential coverage check: only the lines this PR touched
git diff origin/main...HEAD -- '*.py' | \
  diff-cover coverage.xml --compare-branch=origin/main \
  --fail-under=90

diff-cover (Python) or istanbul check-coverage with a --include glob (Node) gives you branch coverage scoped to the changeset. An agent PR that ships 200 lines of new logic with 45% branch coverage on those lines should not merge. A human PR with the same profile probably should not either, but agents hit this failure mode far more often.

Pair this with a mutation score spot-check on the hottest new paths. Running full mutation testing is too slow for CI. Running mutmut or stryker against only the files touched by the PR is feasible in under two minutes on most codebases and will surface tests that pass despite the logic being wrong.

Stage 3: Static Behavioral Analysis (Not LLM Review)

Here is where I will lose some people: I do not run an LLM reviewer in my pipeline at stage 3. I run a static analyzer.

For security properties, tools like semgrep with a custom ruleset, bandit for Python, or CodeQL for multi-language repos catch real vulnerability classes with near-zero false negatives for the patterns they target. An agent writing SQL construction via string interpolation, using pickle.loads on untrusted input, or calling subprocess.shell=True on a user-supplied argument will be caught by these tools deterministically. A GPT-4o reviewer asked to "check for security issues" will catch some of these and miss others in unpredictable ways.

Build a semgrep ruleset specific to your codebase. It takes a day. It pays off immediately on agent PRs because agents pattern-match to the internet's training data, which includes plenty of insecure patterns.

If you want a reference before you build your own, the verification approach OpenThunder takes is worth reading: static, dynamic, and behavioral checks layered in that order.

Stage 4: Where AI Review Actually Earns Its Place

LLM-based review is not useless. It is misplaced at stages 1 through 3.

The right place for an AI reviewer is after all deterministic and static checks pass. At that point, you are not using it to catch structural violations. You are using it for two specific tasks:

  1. Naming and abstraction legibility: agents frequently produce correct code with terrible names. An LLM reviewer is good at flagging processData in a domain context where reconcileTransactionLedger is what the function actually does.
  2. Cross-file consistency: does the agent's new UserService method follow the same error-handling pattern as the other 12 methods in that class? A model with the full file in context can catch that. Static analysis mostly cannot.

Use claude-3-5-sonnet or gpt-4o here with a tightly scoped prompt. Do not ask "review this PR." Ask: "Given the conventions in the surrounding file, flag any method in the diff that handles errors differently from the existing methods. Return only line numbers and a one-sentence reason."

Scope matters. Broad review prompts produce broad, low-confidence output. Narrow prompts produce actionable findings.

The Stage That Is Pure Theater

Skip agent-generated PR descriptions as a verification input. Agents are very good at writing plausible summaries of what they intended to do, not what the diff actually does. Using the PR description to validate the PR is circular and dangerous.

Skip "AI confidence scores" as a gate too. Every model will report high confidence on code it hallucinated the logic for. Confidence scores are self-reported by the system you are trying to verify. They are not a gate; they are a liability.

The strongest counter-argument to this whole pipeline is: "This is too much overhead for a PR that might just be changing a button label." That is a real objection. The answer is to scope your checks by diff size and file path. A PR that touches only *.css or *.md files does not need mutation testing. A PR that touches src/payments/ or db/migrations/ runs the full stack. Your CI config should express that explicitly, not implicitly.

Putting the Stages in Order

To be concrete about sequencing:

  1. Deterministic policy gates: type check, policy lint, license scan, migration safety. Block on any failure.
  2. Differential behavioral coverage: 90% branch coverage on changed lines. Block below threshold.
  3. Static security analysis: semgrep, CodeQL, or equivalent. Block on high/critical findings.
  4. Scoped LLM review: naming, consistency, abstraction legibility. Post as comments, do not auto-block.
  5. Human reviewer: sees a PR that has already passed four automated stages and arrives with specific LLM-flagged questions, not a blank page.

The human reviewer is not removed from the loop. They are moved to where they add irreplaceable value: judgment calls about product intent, architectural direction, and tradeoffs the pipeline cannot score. OpenThunder surfaces the output of stages 1 through 3 as structured findings so that human review time is spent on what only humans can decide.

Put a verification gate on your pipeline

If you are shipping agent-authored PRs into production without a layered verification pipeline, you are betting your incident rate on the agent's training data being good enough. OpenThunder runs static, dynamic, and behavioral checks on every change and turns failures into fixable findings. Try it here.

The difference between a useful agent pipeline and a liability is not whether you review the code; it is whether your pipeline is shaped to the failure modes agents actually have.