Test your CI wrapper with a child process that fails

Imagine a test process that prints a failure message and exits with code 7, but its log-capturing wrapper displays the message and exits with code 0. Test the wrapper from another process with known exit codes and output, because correct failure logs do not guarantee a failing status.

Log capture can hide a failed process

Python's subprocess.run() returns a CompletedProcess object. With its default check=False, a nonzero child exit code does not raise an exception. The caller must inspect or propagate returncode.

Save this deliberately broken wrapper as broken_wrapper.py:

import subprocess
import sys


def main():
    result = subprocess.run(
        sys.argv[1:],
        capture_output=True,
        text=True,
    )
    sys.stdout.write(result.stdout)
    sys.stderr.write(result.stderr)
    # The child's return code is discarded.


if __name__ == "__main__":
    main()

Assuming the child launches and the output writes succeed, this script reaches the end normally and exits with code 0. It does that even when the child exits with code 1 or 7.

Failure text is evidence for a reader. Exit status is the signal to the caller.

Code 7 distinguishes failure detection from status preservation

A wrapper that converts every nonzero result to 1 detects failure but discards the original status. Code 7 tests that distinction.

For a transparent wrapper, I would preserve the child's normal exit status while making its output available. This exit-code matrix gives the required behavior and the expected results for the broken wrapper above:

Some wrappers intentionally normalize all failures to one code. If that is your documented interface, test that mapping instead. For a wrapper whose job is simply to capture logs, preserving the original code avoids silently discarding information.

Correct the wrapper without losing its logs

Save the corrected version as wrapper.py:

import subprocess
import sys


def main():
    result = subprocess.run(
        sys.argv[1:],
        capture_output=True,
        text=True,
    )
    sys.stdout.write(result.stdout)
    sys.stderr.write(result.stderr)
    return result.returncode


if __name__ == "__main__":
    raise SystemExit(main())

The important change is the path from result.returncode to the wrapper's own exit status. Returning a value from main() alone would not set the process exit code; SystemExit makes that connection.

Using check=True alone would not meet the exact-propagation contract either. An uncaught CalledProcessError makes the wrapper fail, but it does not automatically make the wrapper exit with the child's original code.

Start with the failure case

A unit test that mocks subprocess.run() can check a branch inside the wrapper. This fixture checks a different boundary: the status and output visible to the wrapper's caller.

Save this as verify_wrapper.py, beside the other two files:

import argparse
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest


CHILD = """
import sys

code = int(sys.argv[1])
print(f"fixture stdout: {code}")
print(f"fixture stderr: {code}", file=sys.stderr)
sys.exit(code)
"""


class WrapperContract(unittest.TestCase):
    wrapper_path = Path(__file__).with_name("wrapper.py").resolve()

    def test_normal_exits_and_output(self):
        with tempfile.TemporaryDirectory() as workdir:
            for code in (0, 1, 7):
                result = subprocess.run(
                    [
                        sys.executable,
                        str(self.wrapper_path),
                        sys.executable,
                        "-c",
                        CHILD,
                        str(code),
                    ],
                    cwd=workdir,
                    capture_output=True,
                    text=True,
                    timeout=10,
                )

                checks = (
                    ("exit", result.returncode, code),
                    (
                        "stdout",
                        result.stdout,
                        f"fixture stdout: {code}\n",
                    ),
                    (
                        "stderr",
                        result.stderr,
                        f"fixture stderr: {code}\n",
                    ),
                )
                for name, actual, expected in checks:
                    with self.subTest(child_exit=code, check=name):
                        self.assertEqual(actual, expected)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "wrapper",
        nargs="?",
        default=str(WrapperContract.wrapper_path),
    )
    args = parser.parse_args()
    WrapperContract.wrapper_path = Path(args.wrapper).resolve()
    unittest.main(argv=[sys.argv[0]])

python verify_wrapper.py broken_wrapper.py
python verify_wrapper.py wrapper.py

Use python3 instead if that is your interpreter command. The broken wrapper should fail exactly two checks: the exit-status assertions for codes 1 and 7. Its output checks and code-0 status check should pass. The corrected wrapper should satisfy all checks.

The fixture needs only Python's standard library. It uses the current Python interpreter for both subprocess layers, passes arguments without a shell, and runs the wrapper in a temporary working directory. It needs no application, external service, credentials, or customer data.

Exact output assertions catch a wrapper that preserves status but drops or alters one stream. The verifier itself exits unsuccessfully when an assertion fails, so it can serve as a CI check for changes to log capture, report generation, or agent-facing formatting.

The ten-second timeout bounds wrapper execution for this simple fixture; it is not a performance target or a process-tree cleanup guarantee. A wrapper that launches descendants requires separate timeout and cleanup handling.

Why this catches the regression

verifier → wrapper → deterministic child

The child supplies a known status and known output on each stream. The verifier observes the wrapper from outside, just as another process would. With no application logic involved, a failed assertion narrows the investigation to process handling.

This contract covers codes 0, 1, and 7, not signal termination, child-launch errors, cancellation, or timeout handling. Add separate cases for those conditions when your wrapper defines their behavior.

The example also buffers all output in memory and writes it after the child finishes. It is suitable for this small fixture, not a recommendation for handling unlimited test logs. Streaming output and preserving ordering across stdout and stderr require additional design and tests.

Test the hosted CI invocation separately

A passing wrapper test does not prove the pipeline invokes it correctly.

A job can still invoke a different file, ignore a failed command, or place another layer between the wrapper and the final job status. Check these boundaries:

Today, adapt the verifier to your real wrapper's documented interface. Then separately confirm that a disposable invocation with the deterministic failing child makes the hosted job fail. Keep that check separate from production actions: the local verifier cannot establish the hosted job's result.