Make CI reject a missing regression test

For example, a one-unit price test passes while discovery omits the three-unit regression that would expose a bug. For a small set of critical regressions, require their exact discovered test IDs before running that same suite.

A green exit and a stable test count cannot establish that those particular tests were included.

Build a fixture that hides a failing regression

This fixture uses only Python’s standard library. Create a new, isolated directory:

mkdir -p ci-discovery-demo/tests
cd ci-discovery-demo
python3 -c "from pathlib import Path; Path('tests/__init__.py').touch()"

Add the following files. Leave tests/__init__.py empty so that tests is an importable package with stable module-qualified test IDs.

ci-discovery-demo/
├── app.py
├── check_tests.py
└── tests/
    ├── __init__.py
    ├── test_smoke.py
    └── regression_case.py

In app.py, deliberately ignore the quantity:

def total_cents(unit_cents, quantity):
    return unit_cents

In tests/test_smoke.py, exercise the case that happens to work:

import unittest

from app import total_cents


class SmokeTests(unittest.TestCase):
    def test_one_unit(self):
        self.assertEqual(total_cents(125, 1), 125)

In tests/regression_case.py, express the intended behavior for multiple units:

import unittest

from app import total_cents


class TotalTests(unittest.TestCase):
    def test_multiple_units(self):
        self.assertEqual(total_cents(125, 3), 375)

The mismatch is the filename. regression_case.py does not match the explicit test_*.py pattern used below.

Run ordinary discovery from the fixture root:

python3 -m unittest discover -s tests -t . -p 'test_*.py' -v

Expected outcome: only test_one_unit runs, and the command exits with status 0. The multiple-unit assertion would fail if executed, but this command never discovers it.

A test file can exist without participating in CI.

Check required IDs, then execute that same suite

A unittest test ID includes its module, class, and method. Once the regression file has its intended discoverable name, the ID will be:

tests.test_regression.TotalTests.test_multiple_units

Save this guard as check_tests.py:

from pathlib import Path
import sys
import unittest


ROOT = Path(__file__).resolve().parent
REQUIRED_IDS = {
    "tests.test_regression.TotalTests.test_multiple_units",
}


def iter_cases(suite):
    for item in suite:
        if isinstance(item, unittest.TestSuite):
            yield from iter_cases(item)
        else:
            yield item


def main():
    loader = unittest.TestLoader()
    suite = loader.discover(
        start_dir=str(ROOT / "tests"),
        pattern="test_*.py",
        top_level_dir=str(ROOT),
    )

    if loader.errors:
        print("Test loading failed:", file=sys.stderr)
        for error in loader.errors:
            print(error, file=sys.stderr)
        return 2

    discovered_ids = {case.id() for case in iter_cases(suite)}
    missing = REQUIRED_IDS - discovered_ids

    if missing:
        print("Required tests were not discovered:", file=sys.stderr)
        for test_id in sorted(missing):
            print(f"  {test_id}", file=sys.stderr)
        return 2

    result = unittest.TextTestRunner(verbosity=2).run(suite)
    return 0 if result.wasSuccessful() else 1


if __name__ == "__main__":
    sys.exit(main())

The guard discovers one suite, inspects it, then executes that same suite. This avoids checking one selection and accidentally running another. It also rejects test-loading errors rather than accepting an incomplete load.

“Before execution” means before test methods run. Python discovery imports test modules, so this is not an isolation boundary for import-time code.

Run the guard while the regression still has the wrong filename:

python3 check_tests.py

Then rename the regression file and run it again:

mv tests/regression_case.py tests/test_regression.py
python3 check_tests.py

Finally, correct app.py:

def total_cents(unit_cents, quantity):
    return unit_cents * quantity

Run the guard once more:

python3 check_tests.py

These steps distinguish four states:

The assertion failure after the rename is useful: it confirms that the intended regression participates in the command CI will run.

Run the failure cases individually in your terminal. This sequence demonstrates the guard; it is not a sequence for the CI job to repeat. In CI, replace the ordinary discovery command with python3 check_tests.py. Retain that single guarded test command, and make its nonzero result stop the job rather than continuing to a later successful command.

Counts, identities, and exit codes protect different things

If one required test disappears while an unrelated test is added, the count stays unchanged.

A stable count can hide a changed suite.

Use each check for the property it establishes:

A required test marked as skipped still has an ID. If execution is mandatory, add a separate policy for skipped required tests.

Also verify that your CI wrapper preserves the Python process’s nonzero status. A correct guard cannot help if a surrounding script discards its failure.

Keep the discovery contract small and reviewable

I would require IDs for explicitly protected regressions rather than immediately pinning every test in a large suite. Pinning every ID creates maintenance work whenever tests are legitimately renamed or reorganized.

Treat changes to the required set as changes to the protection itself. A rename should update the file and the contract together, after confirming that the same behavior remains covered. Removing an entry from REQUIRED_IDS deserves explicit review: the guard cannot protect a regression if the same change silently deletes its requirement.

A five-minute discovery check

For an existing unittest case, run verbose discovery with the same start directory, top-level directory, and filename pattern as CI. In this fixture, that is:

python3 -m unittest discover -s tests -t . -p 'test_*.py' -v

Read the module, class, and method from the verbose output, then record the exact ID in module.Class.method form. If the case is absent, resolve discovery before adding its ID to the guard. Match the guard’s discovery settings to the CI selection you just checked.

Use this discovery checklist on one important regression:

• Identity: Record its exact runner-visible test ID in REQUIRED_IDS.
• Selection: Confirm that the guard discovers that ID and executes the same suite it inspected.
• Rejection: In an isolated copy, rename the file outside the discovery pattern and confirm that the guard fails.
• Propagation: Confirm that the nonzero status makes the CI job fail.