Most LLM eval suites test what a model says. Almost none test what a model refuses to say, or how that refusal pattern shifts between model versions. That gap is exactly what the Associated Press documented when it probed major AI systems across politically sensitive topics and found that refusal rates, hedging language, and topic avoidance varied in ways that correlated with jurisdiction and training pressures, not with any specification any engineering team had reviewed or approved.
This is not a policy story. It is a regression story. A model that previously answered a category of query and now deflects, softens, or silently omits content has changed its output contract. If you did not write a test for the old behavior, you cannot detect the new behavior. And if you cannot detect it, you will ship it.
Refusal Asymmetry Is a Behavioral Regression Class
The AP study found that some chatbots would discuss historical atrocities in detail depending on which government or group was the subject, generating full answers for one framing and refusing or hedging for a structurally identical prompt with a different actor substituted in. That is not a values decision baked into a published content policy. It is an asymmetric behavioral pattern, and it is exactly the shape of a regression: a function that returns different outputs for logically equivalent inputs based on an undocumented internal state change.
Verification engineers already have a concept for this. Interface invariants. If your API contract says "given a well-formed query about topic X, return a substantive response," then a model update that returns a refusal for some instances of X without updating the contract is a broken invariant. The fact that the breakage is political in nature does not make it less of an engineering artifact. It makes it more dangerous, because the social awkwardness around the subject is exactly what causes teams to leave it untested.
The word refusal asymmetry is worth pinning precisely: it describes cases where a model applies a refusal policy to a subset of a prompt class that is structurally equivalent to prompts it handles freely. One framing gets an answer. A mirrored framing gets a hedge or a blank. No release note. No contract update. No CI failure.
That is a silent regression. You own it the moment it reaches production.
How to Spec the Contract Before You Test It
You cannot gate what you have not defined. The first step is writing refusal behavior into your model interface spec the same way you would write expected status codes into an API spec.
Start with prompt equivalence classes. A prompt equivalence class is a set of semantically distinct prompts that your contract says should receive the same treatment: either all answered, all refused, or all routed to a human. The AP study's methodology is a good template here. Take a sensitive topic. Write a canonical prompt. Now write five structural mirrors of that prompt: same question type, same level of specificity, different actor, different geography, different historical period. Your contract should state whether this class is in-scope for response, and if it is, all five mirrors should return substantive answers or all five should refuse. Asymmetric outcomes across the class are a contract violation, regardless of which outcome you prefer.
Document three things per class:
- The canonical prompt template, with clearly marked substitution variables.
- The expected response disposition: answer, refusal, or partial answer with caveats.
- The classifier threshold you will use to label a real model response as passing or failing that disposition.
The third point is where most teams stop too early. They write prompts but never write classifiers, so they end up eyeballing outputs during review. That does not scale past a handful of prompts and it does not run in CI.
For answer versus refusal, a lightweight classifier works well: check for explicit refusal phrases ("I can't help with that", "I'm not able to discuss", "This topic is outside"), check response length against a minimum threshold for the prompt class, and check for hedge density using a short phrase list ("it's complicated", "perspectives vary", "I'd encourage you to"). A response that triggers two or more of those signals is a candidate refusal regardless of whether it contains any actual content. You do not need a fine-tuned model for this. A rules-based classifier with a small validation set is auditable, fast, and does not introduce a second model's own refusal biases into your pipeline.
For asymmetry detection specifically, run your full equivalence class in a single eval job and compute the refusal rate across mirrors. A refusal rate above zero but below 100 percent is the signal. Uniform refusal across a class might be a policy decision you disagree with, but it is at least consistent. Non-uniform refusal is a contract violation on its face.
Building the Eval Suite and Wiring It to CI
Here is the practical shape of a refusal-behavior eval suite that a team of two can build in a week and maintain indefinitely.
Prompt corpus. Keep your prompt equivalence classes in a versioned flat file, one row per prompt with columns for class ID, template, substitution values, and expected disposition. Version this file in your model repository alongside your model config. When a model version changes, the prompt corpus version must also be reviewed. That review is your contract amendment process.
Runner. Write a thin eval runner that takes a batch of prompts, calls your model endpoint (Claude, GPT-4o, a self-hosted Llama variant, whatever you are running), and writes raw completions to a results file keyed by prompt ID. No post-processing in the runner. Keep it dumb. Post-processing belongs in the classifier.
Classifier. The classifier reads the results file and applies your disposition rules. Output a per-prompt pass or fail, a per-class symmetry score (fraction of mirrors with consistent disposition), and a top-level summary: classes passing, classes failing, new failures since last run. Store this output as a structured artifact.
Gate. The CI gate compares the current artifact against the baseline artifact from your last approved model version. Any class that moves from passing to failing is a blocking failure. Any class with a symmetry score that drops by more than your configured threshold is a warning. You decide the threshold. I would start at 0.15, meaning if more than 15 percent of mirrors in a class change disposition, that is a gate failure.
This is the same pattern you use for output conformance checks on structured generation, and the tooling transfers directly. The only LLM-specific addition is the symmetry scoring step.
Run your refusal eval suite against every model version bump, every system prompt change, and every fine-tuning run. Do not assume that a change scoped to capability X cannot affect refusal behavior for topic Y. Fine-tuning is global, not local. A model tuned to be more helpful on coding tasks can shift its calibration on sensitive topics as a side effect. You will not know unless you test.
Test across temperature settings too. A model that answers freely at temperature 0 and refuses at temperature 0.7 has a non-deterministic contract, and you need to know that before your users find it. Run each prompt five times at your production temperature and flag any prompt where the disposition is not consistent across runs. Stochastic refusal is worse than deterministic refusal because it is invisible to any single-sample eval.
Keep your baseline artifacts. When a refusal regression surfaces in production, you will want to know exactly when the contract changed and which model version introduced it. Artifact history is your audit trail. Without it, you are doing forensics without evidence.
OpenThunder's behavioral check layer is built around exactly this pattern: versioned baselines, classifier-gated comparisons, and blocking failures surfaced as fixable findings rather than noise. If you are wiring this up from scratch, the OpenThunder pipeline model gives you the infrastructure so you can focus on writing the prompt corpus and classifiers rather than building a CI harness.
The AP findings are not an anomaly. They are a preview of what silent model drift looks like at scale: gradual, asymmetric, and completely invisible to any team that does not have a written contract for refusal behavior. The solution is not to have opinions about what the model should refuse. The solution is to have a spec, a test suite, and a gate. Once you have those, whether the model is drifting toward more refusals or fewer, you will know within one pipeline run.
Put a verification gate on your pipeline
If your current CI setup has no behavioral checks on model output, you are shipping on trust. OpenThunder runs static, dynamic, and behavioral checks on every change and turns failures into fixable findings. Try it here.
A model without a tested refusal contract is not a product decision: it is an untested assumption waiting to become a production incident.