Minimize a Regression Test Without Losing the Original Failure

Summary slide. Accept a smaller input only if the baseline passes and the changed code fails the same named assertion. Artwork is AI-generated; every word on the slide comes from this article. Save the slide (PNG)

An illustrative configuration fixture assigns retries twice: the baseline correctly keeps 3, while a refactored parser incorrectly keeps 1. Shrinking that fixture to just retries makes both parsers fail, so the smaller input no longer demonstrates the regression.

Accept a smaller input only when the baseline still passes and the changed implementation fails the same named assertion, without unrelated execution failures.

A smaller crash is not necessarily a smaller reproducer.

Separate reduction from permanent verification

Use an input reducer when you have a known failing fixture but cannot yet see which parts are necessary, whether the change came from an AI assistant or a human author. Reduction preserves the failure; a permanent CI test must require the corrected implementation to pass.

For this illustrative parser, the behavior is precise:

• Blank lines and full-line comments are ignored.
• Each remaining line contains a key and value separated by =.
• Repeated keys use the last assigned value.
• The fixture must end with an effective retries value of 3.

The refactor changes ordinary assignment to setdefault, accidentally making the first value win. The reducer's job is to preserve that distinction.

Save the three-gate predicate

Call this the same-failure gate. Every candidate must pass all three checks:

Check candidates against the gate

These are expected outcomes from the example's control flow, not measured execution results. \n represents a newline.

The empty first value matters. This illustrative parser permits empty values, so retries=\nretries=3\n still demonstrates first-assignment versus last-assignment behavior. If the real format forbids empty values, encode that rule in the baseline or in an explicit validity check. Do not assume the reducer knows your input grammar.

The malformed candidate retries shows the exit-code trap. Both parsers fail while splitting a line without =. A predicate that accepts any nonzero exit from the changed parser would accept it, despite the missing behavioral comparison.

An exit code reports failure, not failure identity.

The three gates are not proof that every accepted input has the same internal cause. A named assertion identifies an observed failure, not a complete causal explanation. For this small parser, inspecting the two assignment statements makes the cause clear. For a larger system, you may need a more specific assertion or additional output checks.

Build an isolated, dependency-free example

The following illustrative script uses only the Python standard library. Paste the next three Python blocks, in order, into reduce_config.py to make the complete script.

It keeps both parser versions in one file for clarity. Each candidate runs in a fresh subprocess against a temporary local file. No production resources or credentials are involved. This is a runnable example, not a reported test run.

1. Define the worker protocol

The worker distinguishes a passing assertion, the named failure, and an unrelated error. The parent accepts only exact protocol responses with matching process exit codes; timeouts, crashes, and malformed responses are invalid.

The assertion uses an explicit exception rather than Python's assert statement, so optimized execution cannot remove it.

import json
from pathlib import Path
import subprocess
import sys
import tempfile


TARGET = "retries_last_assignment_wins"
TIMEOUT_SECONDS = 2  # Illustrative local limit; tune for your test.
INITIAL = """# Example configuration
region=west
retries=1
banner=welcome
retries=3
"""


class TargetFailure(Exception):
    pass


def parse_config(text, implementation):
    result = {}
    for raw in text.splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        key, value = line.split("=", 1)
        key, value = key.strip(), value.strip()
        if implementation == "baseline":
            result[key] = value
        else:
            result.setdefault(key, value)  # Deliberate regression.
    return result


def worker(implementation, path):
    try:
        text = Path(path).read_text(encoding="utf-8")
        result = parse_config(text, implementation)
        if result.get("retries") != "3":
            raise TargetFailure(TARGET)
    except TargetFailure:
        payload, code = {"status": "target_failure", "name": TARGET}, 1
    except Exception as exc:
        payload, code = {
            "status": "error", "type": type(exc).__name__
        }, 2
    else:
        payload, code = {"status": "pass", "name": TARGET}, 0
    print(json.dumps(payload))
    return code


def run_worker(implementation, path):
    command = [
        sys.executable,
        str(Path(__file__).resolve()),
        "--worker",
        implementation,
        str(path),
    ]
    try:
        completed = subprocess.run(
            command,
            capture_output=True,
            text=True,
            timeout=TIMEOUT_SECONDS,
        )
        payload = json.loads(completed.stdout)
    except (OSError, subprocess.TimeoutExpired, ValueError):
        return "invalid"

    # Require an exact protocol response and matching process exit code.
    if completed.returncode == 0 and payload == {
        "status": "pass", "name": TARGET
    }:
        return "pass"
    if completed.returncode == 1 and payload == {
        "status": "target_failure", "name": TARGET
    }:
        return "target_failure"
    return "invalid"

2. Make acceptance depend on both implementations

This is the central decision. Reject the candidate unless the baseline passes; only then check whether the changed parser produces the named failure. An unrelated exception cannot qualify.

def preserves_failure(text):
    with tempfile.TemporaryDirectory() as directory:
        path = Path(directory) / "candidate.txt"
        path.write_text(text, encoding="utf-8")
        if run_worker("baseline", path) != "pass":
            return False
        return run_worker("changed", path) == "target_failure"

3. Search by deletion and verify the result

The search first tries deleting groups of lines, then groups of characters. Line deletion removes unrelated settings cheaply; character deletion can expose smaller inputs that line deletion cannot reach.

This is a deletion-only search. It does not promise the globally shortest input, and another search order can produce a different useful reproducer.

def reduce_parts(parts):
    # Delete chunks, then try progressively finer deletions.
    granularity = 2
    while len(parts) >= 2:
        chunk_size = (len(parts) + granularity - 1) // granularity
        for start in range(0, len(parts), chunk_size):
            candidate = parts[:start] + parts[start + chunk_size:]
            if preserves_failure("".join(candidate)):
                parts = candidate
                granularity = max(2, granularity - 1)
                break
        else:
            if granularity >= len(parts):
                break
            granularity = min(len(parts), granularity * 2)
    return parts


def main():
    args = sys.argv[1:]
    if len(args) == 3 and args[0] == "--worker":
        if args[1] not in {"baseline", "changed"}:
            return 2
        return worker(args[1], args[2])

    if len(args) == 2 and args[0] == "--verify":
        status = run_worker("changed", Path(args[1]).resolve())
        print(status)
        return 0 if status == "pass" else 1

    if args:
        print("Usage: python reduce_config.py [--verify FILE]")
        return 2

    if not preserves_failure(INITIAL):
        print("Initial fixture does not satisfy the same-failure gate.")
        return 1

    lines = reduce_parts(INITIAL.splitlines(keepends=True))
    reduced = "".join(reduce_parts(list("".join(lines))))
    if not preserves_failure(reduced):
        print("Final verification failed; no reproducer written.")
        return 1

    output = Path("minimized.txt")
    output.write_text(reduced, encoding="utf-8")
    print(f"Wrote {output}: {reduced!r}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run the reducer with Python 3:

python reduce_config.py

The output path is a local example artifact. The script overwrites minimized.txt, so run it in a scratch directory.

Use the minimized fixture as a passing CI obligation

Before fixing the parser, this command should report target_failure and exit unsuccessfully:

python reduce_config.py --verify minimized.txt

Correct the changed implementation by replacing:

result.setdefault(key, value)

with:

result[key] = value

The same verification command should then report pass and exit successfully. Those are expected results to check locally, not claimed execution results.

Commit the fixture and the passing verification check. In a real repository, have the verifier import the production parser rather than retaining a separate copy of it. Otherwise, CI could verify the demonstration while the actual implementation regresses.

Reduction preserves a failure. CI should prevent its return.

Keep the baseline available for reduction if useful, but do not require the regression to remain present in routine CI. After the fix, the reducer should refuse its original fixture because the changed parser no longer fails the target assertion.

Static analysis can complement this check, but this example's contract concerns the value produced from repeated assignments. Verify that behavior directly. If the defect depends on loading files, environment settings, or another component, preserve that relevant boundary in an isolated integration test rather than reducing it away.

Strengthen the predicate when the evidence is less certain

This example is deterministic and has one relevant assertion. Its two-second timeout is a local example setting, not a general recommendation.

Adapt the same-failure gate when your system has additional uncertainty:

• Multiple assertion failures: identify the target by a stable test name and, where necessary, structured output details.
• Nondeterministic results: establish a repeatability policy before reducing. One matching failure may be a coincidence.
• External dependencies: use isolated test resources and distinguish unavailable infrastructure from the target behavior.
• Expensive execution: begin with coarse reduction or a smaller isolated harness. Character-level searches can require many subprocesses.

Timeouts and unrelated errors are rejected here, not diagnosed. If the starting fixture cannot pass the gate, repair or investigate the harness before trusting any reduction.

For a five-minute exercise, run the script in a scratch directory, inspect minimized.txt, and manually compare retries with retries=1\nretries=3\n. Then fix the assignment and run --verify. That exercise demonstrates the predicate's boundary; it does not validate a real application's parser.

Save the field guide

A field guide distilled from this article. The text version follows below.

Save the infographic (PNG)

Text version

• The baseline no longer passes: The smaller input fails the baseline's expected-result assertion or raises an error. Reject it because it no longer preserves a valid comparison with the changed implementation.
• The changed code fails differently: The changed implementation passes or produces an unrelated exception instead of the named assertion failure. Reject it because a nonzero exit alone does not identify the regression.
• The run cannot be trusted: A run times out, crashes, or returns malformed output. Reject the candidate because both runs must finish within the limit and return recognized results.
• The fix is ready for CI: The corrected implementation passes verification on the minimized fixture. Commit the fixture and a passing check that imports the production parser so CI tests the actual implementation.

Illustrative parser comparison: For retries=1\nretries=3\n, the baseline is expected to pass while the changed parser fails retries_last_assignment_wins. Reducing it to retries makes both parsers raise ValueError, so reject that candidate.

A named assertion identifies an observed failure, not its complete cause. Deletion-only reduction also does not guarantee the globally shortest input.