Test AI Shopping Eligibility With an Exact-List CI Assertion

Catch unknown IDs, missing availability, and nonboolean values with an isolated Python test that proves eligible products survive in order.

Summary slide: Unknown is not available Require explicit permission. Test missing data. Assert the whole list.

Summary slide. Use an exact expected list to prove your filter keeps eligible products and rejects everything else. Save the slide (PNG)

A filter can reject explicitly unavailable products while still accepting unknown IDs and records with missing availability. In this illustrative four-candidate example, only cap-available should survive:

Candidate ID	Catalog record	Expected decision
`cap-available`	`{"available": true}`	Keep
`jersey-unavailable`	`{"available": false}`	Reject
`scarf-missing`	`{}`	Reject
`unknown-product`	No record	Reject

The exact expected output is:

["cap-available"]

Test that boundary by passing fixed candidate IDs through your eligibility function against a versioned catalog fixture, then asserting the exact allowed output. The assertion must prove both rejection and retention.

Let the model propose candidates, but enforce catalog eligibility with deterministic code. That makes an eligibility regression reproducible without a live model call, a changing catalog, or an engagement experiment.

Define what counts as eligible

Assume this example service accepts an ordered list of canonical string product IDs and a catalog snapshot. Its contract is:

An ID must exist in the supplied catalog.
Its available field must be the boolean True.
Missing availability data means ineligible.
Eligible IDs retain their input order.

This is a deliberately narrow contract. It does not address duplicate IDs, regional restrictions, pricing, or inventory reservations.

Unknown is not available.

That sentence becomes executable only when the behavior for absent records and fields is explicit.

Reproduce the fail-open bug

A faulty filter might look like this:

def eligible_ids(candidate_ids, catalog):
    return [
        product_id
        for product_id in candidate_ids
        if catalog.get(product_id, {}).get("available", True)
    ]

The two defaults interact badly:

An unknown ID becomes an empty record.
An empty record defaults to available.

For the fixture above, this implementation would return:

["cap-available", "scarf-missing", "unknown-product"]

This is a prediction from the code, not a reported test run. The unavailable jersey is rejected, so a test containing only known available and unavailable products would miss the defect.

A negative fixture needs absent data, not just false data.

Put the corrected function behind an isolated assertion

The following example uses only Python’s standard library. Save both files in the same directory. It requires no credentials, network access, model endpoint, or live customer data.

First, put the corrected filter in recommendation_eligibility.py:

def eligible_ids(candidate_ids, catalog):
    return [
        product_id
        for product_id in candidate_ids
        if product_id in catalog
        and catalog[product_id].get("available") is True
    ]

The membership check rejects unknown IDs. The strict boolean check rejects missing values and nonboolean values such as 1 and the string "true", which this example contract does not accept. This assumes catalog records are dictionaries; malformed record types belong in a separate schema-validation test.

Next, create test_recommendation_eligibility.py. Keep the four-case fixture as the baseline, then add focused tests for order preservation and strict boolean availability:

import unittest

from recommendation_eligibility import eligible_ids


class RecommendationEligibilityTest(unittest.TestCase):
    def test_only_explicitly_available_products_survive(self):
        # Versioned snapshot: no live catalog reads.
        catalog = {
            "cap-available": {"available": True},
            "jersey-unavailable": {"available": False},
            "scarf-missing": {},
        }
        candidates = [
            "cap-available",
            "jersey-unavailable",
            "scarf-missing",
            "unknown-product",
        ]

        self.assertEqual(
            eligible_ids(candidates, catalog),
            ["cap-available"],
        )

    def test_multiple_eligible_ids_keep_candidate_order(self):
        catalog = {
            "cap-available": {"available": True},
            "jersey-unavailable": {"available": False},
            "scarf-available": {"available": True},
        }
        candidates = [
            "scarf-available",
            "jersey-unavailable",
            "cap-available",
        ]

        self.assertEqual(
            eligible_ids(candidates, catalog),
            ["scarf-available", "cap-available"],
        )

    def test_nonboolean_availability_is_rejected(self):
        for value in (1, "true"):
            with self.subTest(availability=value):
                catalog = {
                    "cap-available": {"available": True},
                    "nonboolean-product": {"available": value},
                }

                self.assertEqual(
                    eligible_ids(
                        ["nonboolean-product", "cap-available"],
                        catalog,
                    ),
                    ["cap-available"],
                )


if __name__ == "__main__":
    unittest.main()

The baseline cannot verify order preservation with only one eligible ID. The second test supplies two eligible IDs in an order different from their catalog insertion order. The third rejects both numeric and string stand-ins for True, including 1, which compares equal to True in Python but is not the boolean True.

Run the same command locally and in the CI job that runs your Python checks:

python -m unittest -v test_recommendation_eligibility

The expected result is passing assertions with the corrected implementation. To reproduce the regression in an isolated working copy, replace the function body with the faulty version and rerun the command. The baseline assertion should fail, showing the two extra IDs; the nonboolean test should fail too. These examples have not been executed here.

The exact-list assertion matters. Checking only that every returned ID exists in the catalog would miss scarf-missing. Checking only that the output has one item could accept the wrong product. Checking only for rejected IDs could allow a filter that returns nothing.

The expected output must prove both rejection and retention.

For a real application gate, import the function used by the recommendation response path. Testing a copied helper proves only that the copy behaves correctly.

Use the Four-Case Eligibility Gate in code review

Keep this checklist beside the test when reviewing changes. Each case targets a different implementation error:

Signal	Required behavior	Implementation error it catches
Positive eligibility	Known ID with explicit `True` survives	Rejecting everything can satisfy rejection-only checks
Explicit exclusion	Known ID with explicit `False` is removed	Checking catalog membership without checking availability
Incomplete evidence	Known ID with a missing field is removed	Defaulting a missing availability field to available
Missing identity	Unknown ID is removed	Treating an absent record as an eligible empty record

This is the Four-Case Eligibility Gate. It targets the boundary where a candidate becomes an accepted recommendation, not the complete contract suite. Keep the order and nonboolean tests alongside it.

I would use this deterministic gate for rules whose inputs and expected decisions can be specified exactly. Under the example contract, availability belongs here. If a service also has regional eligibility rules, those need their own explicit inputs and fixtures rather than assumptions hidden inside this test.

I would not use this test to decide whether a cap matches a jersey, whether the ranking is useful, or whether shoppers will engage. Eligibility asks whether a product is allowed under the catalog contract; quality asks whether it is useful for the shopper.

In How Fanatics and OpenAI streamline the fan experience, OpenAI says Fanatics uses OpenAI for Shop the Look to help fans complete game-day outfits, and reports 10% higher engagement from testing across roughly 30,000 MLB products. That publisher-reported engagement result does not establish catalog correctness. The supplied excerpt does not describe Fanatics’ architecture, eligibility rules, or testing methods; this instructional example is not a reconstruction of its system.

Know what the green check leaves uncertain

These checks verify the filter’s behavior for fixed candidate lists and catalog snapshots. They do not establish that:

The live catalog contains correct or fresh data.
The response handler actually invokes this filter.
A model response is parsed into candidate IDs correctly.
Availability remains unchanged between recommendation and checkout.
The recommendations are relevant or engaging.

Add a separate integration check for the response path to verify that rejected IDs cannot bypass the filter. Keep that check isolated too: use a fake candidate provider and a test catalog, not a live model and production inventory.

A frozen fixture makes a failure reproducible, not a catalog current.

Try the regression in five minutes

Create the two files, run the assertions, then introduce the faulty default in an isolated working copy. Confirm that the tests detect the regression before restoring the corrected implementation.

This exercise checks one contract at one boundary. It does not establish catalog freshness or recommendation quality.

In your application, run the assertions against the production eligibility function, then separately verify that the response path cannot bypass it.

Save the field guide

Field guide: Build an exact-list eligibility gate

A field guide distilled from this article. The text version follows below.

Save the infographic (PNG)

Text version

Define permission: A candidate may be unknown, unavailable, or missing availability data. Require catalog membership and boolean True availability, because missing evidence does not establish eligibility.
Build the baseline: Tests with only available and unavailable products can miss defaults that admit missing data. Use a versioned fixture with available, unavailable, missing-field, and unknown cases to expose those defaults.
Check the exact output: A count or rejection-only check can pass while returning the wrong products or nothing. Assert the full expected list, then add order and nonboolean tests because the baseline does not cover the whole contract.
Connect the real response path: A passing helper test does not prove recommendations pass through the filter. Test the production eligibility function and separately check the response path with a fake provider and test catalog to prevent bypass.

Illustrative four-candidate check: Of cap-available, jersey-unavailable, scarf-missing, and unknown-product, only cap-available should survive. The exact expected list is ["cap-available"].

Fixed fixtures verify filter behavior, not catalog freshness or recommendation quality. A separate integration check is needed to verify that the response path cannot bypass the filter.