Source-tree tests can pass while the wheel lacks a resource

For example, a package's greeting loader returns Hello from the wheel.\n in the checkout, but its built wheel omits the template. To catch that omission, install the wheel in a fresh environment and verify that the import comes from that environment, not the checkout.

For changes to runtime resources or packaging configuration, I would add a targeted installed-artifact check alongside the ordinary suite. A passing source-tree test does not establish that a wheel contains its runtime resources.

A four-point wheel gate

Use this checklist when reviewing a packaging-sensitive change:

1. Artifact: Is the check installing the wheel intended for release rather than an editable checkout?
2. Origin: Does it verify that imports resolve inside the fresh environment?
3. Behavior: Does it exercise a resource-dependent operation and assert an exact expected result?
4. Failure: Does a deliberately omitted resource make the gate fail?

The last point tests the check itself. Introduce the omission in a clean temporary project copy, without generated build state, so retained files cannot hide it.

Reproduce a resource that exists only in the checkout

This exercise uses Python 3.10 or newer and a POSIX shell. It creates temporary directories rather than modifying an existing project. The build environment needs pip and the build frontend; the declared build backend uses setuptools and wheel. Installing these tools requires access to a package index or an appropriate local cache.

Create a package with one loader and one template:

root="$(mktemp -d)"
checkout="$root/project"
mkdir -p "$checkout/wheel_probe/templates"
cd "$checkout"

cat > pyproject.toml <<'TOML'
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "wheel-resource-probe"
version = "0.1.0"
requires-python = ">=3.10"

[tool.setuptools]
packages = ["wheel_probe"]
include-package-data = false

[tool.setuptools.package-data]
wheel_probe = []
TOML

cat > wheel_probe/__init__.py <<'PY'
from importlib.resources import files


def greeting() -> str:
    return (
        files("wheel_probe")
        .joinpath("templates")
        .joinpath("greeting.txt")
        .read_text(encoding="utf-8")
    )
PY

printf '%s\n' 'Hello from the wheel.' \
  > wheel_probe/templates/greeting.txt

The packaging configuration deliberately disables automatic package-data inclusion and declares no resource files. The Python module is included; the template is not.

Run the checkout check:

python3 -c 'from wheel_probe import greeting; assert greeting() == "Hello from the wheel.\n"'

The expected result is a successful exit because Python can import the package from the current directory, where the template exists. That result says nothing about the wheel's contents.

Make the installed check prove where it imported from

Save a separate check outside the project:

cat > "$root/check_installed.py" <<'PY'
from pathlib import Path
import sys

import wheel_probe

assert sys.prefix != sys.base_prefix, "Run this check in a virtual environment"

origin = Path(wheel_probe.__file__).resolve()
environment = Path(sys.prefix).resolve()
assert origin.is_relative_to(environment), (
    f"Imported {origin}, outside the expected environment {environment}"
)

expected = "Hello from the wheel.\n"
actual = wheel_probe.greeting()
assert actual == expected, f"Expected {expected!r}, got {actual!r}"
PY

The import-location assertion checks that the module resides inside the fresh environment. The final assertion exercises the public loader and checks the exact value, including its trailing newline.

Testing only whether the template exists would miss a loader that reads the wrong file or returns altered text. Testing only whether the function returns a nonempty string would accept incorrect content.

Test the value callers need, not merely the presence of a file.

Build, install, and check without checkout imports

First, create a separate environment for the build frontend:

python3 -m venv "$root/build-env"
"$root/build-env/bin/python" -m pip install build

Then run this block. Every invocation creates a clean build-input copy, a new artifact directory, a runtime environment, and a working directory. For this minimal project, only pyproject.toml and the package directory are needed as build inputs:

(
  set -eu

  build_input="$(mktemp -d "$root/build-input.XXXXXX")"
  artifacts="$(mktemp -d "$root/artifacts.XXXXXX")"
  runtime="$(mktemp -d "$root/runtime.XXXXXX")"
  run_dir="$(mktemp -d "$root/run.XXXXXX")"

  cp "$checkout/pyproject.toml" "$build_input/"
  cp -R "$checkout/wheel_probe" "$build_input/"

  "$root/build-env/bin/python" -m build \
    --wheel --outdir "$artifacts" "$build_input"

  python3 -m venv "$runtime"
  "$runtime/bin/python" -m pip install --no-deps "$artifacts"/*.whl

  cd "$run_dir"
  "$runtime/bin/python" -I "$root/check_installed.py"
)

The expected failure is a FileNotFoundError when the installed loader tries to read the omitted template. That failure demonstrates that the gate catches the intended omission.

Several details prevent an accidental pass:

• A clean build-input copy excludes generated state such as setuptools' build/lib, which can retain files from an earlier build.
• A fresh artifact directory prevents the wheel glob from selecting an older build.
• A fresh runtime environment avoids a previous installation supplying the resource.
• A working directory outside the checkout removes the project directory as an implicit import source.
• Python's -I flag ignores PYTHONPATH and excludes the script directory and user site-packages from the import path.
• The import-location assertion checks that the loaded module actually resides in the runtime environment.

Do not replace the wheel installation with pip install -e .. An editable installation changes what you are testing and can expose checkout files that are absent from the distribution.

The example uses --no-deps because this package has no runtime dependencies. For a real package, explicitly install its required dependencies, ideally from your controlled dependency inputs. An import failure caused by missing dependencies is a different defect from the resource omission being tested here.

Include the resource, then repeat the same check

Change the package-data section in pyproject.toml to:

[tool.setuptools.package-data]
wheel_probe = ["templates/*.txt"]

Run the same build-install-check block again. It creates fresh output and runtime directories, so you do not need to uninstall the broken package or select between old wheels. It also copies the source inputs into a clean build directory: a fresh wheel output directory alone does not clear setuptools' retained build files.

The expected result is a successful exit: the module imports from the fresh environment, and its loader returns exactly Hello from the wheel.\n. Keep the assertion unchanged and correct the artifact's contents instead.

For an existing project, the clean temporary copy must contain all required source inputs but no generated build state. This matters especially when deliberately removing a resource: a retained copy in build/lib can otherwise enter the wheel and conceal the omission.

These commands isolate imports and installations; they do not make the build byte-for-byte reproducible. If you need repeatable tool selection, pin the build frontend, backend, and other build dependencies in your CI tooling policy.

Keep the ordinary suite and add an artifact gate

The useful distinction is which filesystem layout each check exercises:

Keep both: the artifact gate covers a distribution boundary, while the ordinary suite covers behavior beyond that selected resource and environment.

For CI, preserve the commands in a script and make a nonzero exit fail the job. Do not run these checks with -O or -OO, because those flags disable the assertions. Although -I ignores PYTHONOPTIMIZE, explicitly adding optimization flags would still disable the origin and exact-value checks.

Build the wheel once, run the gate against that wheel, and retain the same artifact for any later publication step. Rebuilding after verification creates another artifact whose contents were not checked by that run.

Choose one resource today, then validate the gate

Spend five minutes choosing one existing runtime resource and defining the exact behavior callers should receive from it.

The subsequent validation exercise takes that expectation through installation and failure testing. Build from clean inputs, install the wheel in a disposable environment outside the checkout, and check both import origin and behavior. Then deliberately omit the resource in a clean temporary project copy without generated build state, rebuild into a fresh output directory, and confirm failure in another fresh runtime environment.

Start with the resource boundary your change actually touches. The check's coverage remains limited to the behavior and environment you exercise.