Test per-site segmentation gates with a small fixture whose overall Dice improves while one site's Dice falls. Run it through both acceptance rules and assert that the pooled-only gate passes while the site-aware gate fails.
For example, use four cases: three from site A and one from site B. Give every baseline prediction a Dice score of 0.75, then make the candidate perfect on site A and worse on site B. The overall mean rises to 0.875 even though site B drops to 0.50.
An average cannot enforce a per-site promise.
Build a fixture with two deliberately different outcomes
This fixture tests aggregation logic, not a segmentation model. All masks are one-dimensional binary tuples, standing in for flattened image masks. Each contains eight pixels and exactly four foreground pixels.
Use the same ground-truth mask for every case:
Ground truth: 1 1 1 1 0 0 0 0
Baseline prediction:1 1 1 0 1 0 0 0
Site A candidate: 1 1 1 1 0 0 0 0
Site B candidate: 1 1 0 0 1 1 0 0
For binary masks, define Dice as:
[ D(P,G)=\frac{2|P\cap G|}{|P|+|G|} =\frac{2TP}{2TP+FP+FN} ]
Every prediction here has four foreground pixels, so every denominator is eight:
- The baseline has three overlapping foreground pixels: Dice = 6/8 = 0.75.
- The site-A candidate has four: Dice = 8/8 = 1.00.
- The site-B candidate has two: Dice = 4/8 = 0.50.
The expected case-level results are:
| Case ID | Site ID | Baseline Dice | Candidate Dice | Change |
|---|---|---|---|---|
| A-1 | A | 0.75 | 1.00 | +0.25 |
| A-2 | A | 0.75 | 1.00 | +0.25 |
| A-3 | A | 0.75 | 1.00 | +0.25 |
| B-1 | B | 0.75 | 0.50 | −0.25 |
Define the pooled score here as the arithmetic mean of case-level Dice scores, with each case receiving equal weight:
[ D_{overall}=\frac{1}{N}\sum_{i=1}^{N}D_i ]
Define each site's score as the same mean restricted to that site. Under these definitions:
| Aggregation | Baseline | Candidate | Change |
|---|---|---|---|
| Overall case mean | 0.750 | 0.875 | +0.125 |
| Site A case mean | 0.750 | 1.000 | +0.250 |
| Site B case mean | 0.750 | 0.500 | −0.250 |
For this example, allow an overall regression of at most 0.02 Dice and a regression at any individual site of at most 0.05 Dice. These are illustrative absolute tolerances, not clinical standards or relative percentages.
The pooled-only rule passes because +0.125 is at least −0.02. The site-aware rule fails because site B's −0.250 change is below −0.05.
The larger site's improvements pay for the smaller site's regression in the overall mean. Checking site B separately prevents that trade.
Why this distinction matters for evaluation tooling
The abstract of Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training, hosted on arXiv, reports that adapted DINOv2 generalized better on every external dataset evaluated. It also reports that task-specific models remained competitive within the training distribution when training data were abundant, and delineated boundaries more precisely.
Those are the authors' reported findings, not independently replicated results. For engineers building evaluation pipelines, the useful implication is narrower than choosing a winning model: a comparison can depend on where the evaluation data came from and which property the metric measures.
Preserving site-level results lets a gate enforce a site-level requirement. It does not make Dice a boundary-specific metric, establish generalization to unseen institutions, or turn an evaluation threshold into a deployment safety criterion.
Assert the acceptance decisions, not just the scores
Save the following as test_segmentation_gate.py. It uses only Python's standard library and operates entirely on synthetic, in-memory masks.
The implementation intentionally rejects empty-empty masks rather than assigning them a Dice score. That policy does not affect this fixture, but making it explicit prevents an undefined case from silently changing the evaluator's behavior.
from collections import defaultdict
from dataclasses import dataclass, replace
from statistics import mean
import unittest
@dataclass(frozen=True)
class Case:
case_id: str
site: str
truth: tuple
baseline: tuple
candidate: tuple
TRUTH = (1, 1, 1, 1, 0, 0, 0, 0)
BASELINE = (1, 1, 1, 0, 1, 0, 0, 0)
WORSE = (1, 1, 0, 0, 1, 1, 0, 0)
CASES = (
Case("A-1", "A", TRUTH, BASELINE, TRUTH),
Case("A-2", "A", TRUTH, BASELINE, TRUTH),
Case("A-3", "A", TRUTH, BASELINE, TRUTH),
Case("B-1", "B", TRUTH, BASELINE, WORSE),
)
def dice(prediction, truth):
if len(prediction) != len(truth):
raise ValueError("Mask lengths must match")
if any(v not in (0, 1) for v in (*prediction, *truth)):
raise ValueError("Masks must be binary")
denominator = sum(prediction) + sum(truth)
if denominator == 0:
raise ValueError("Empty-empty masks require an explicit policy")
overlap = sum(p * g for p, g in zip(prediction, truth))
return 2 * overlap / denominator
def evaluate(cases, overall_tolerance=0.02, site_tolerance=0.05):
if not cases:
raise ValueError("Evaluation set must not be empty")
if any(not case.site.strip() for case in cases):
raise ValueError("Every case must have a site label")
if len({case.case_id for case in cases}) != len(cases):
raise ValueError("Case IDs must be unique")
baseline_scores = []
candidate_scores = []
by_site = defaultdict(list)
for case in cases:
baseline = dice(case.baseline, case.truth)
candidate = dice(case.candidate, case.truth)
baseline_scores.append(baseline)
candidate_scores.append(candidate)
by_site[case.site].append((baseline, candidate))
overall_baseline = mean(baseline_scores)
overall_candidate = mean(candidate_scores)
overall_delta = overall_candidate - overall_baseline
site_deltas = {
site: mean(candidate for _, candidate in pairs)
- mean(baseline for baseline, _ in pairs)
for site, pairs in by_site.items()
}
pooled_pass = overall_delta >= -overall_tolerance
site_aware_pass = pooled_pass and all(
delta >= -site_tolerance for delta in site_deltas.values()
)
return {
"overall_baseline": overall_baseline,
"overall_candidate": overall_candidate,
"overall_delta": overall_delta,
"site_deltas": site_deltas,
"pooled_pass": pooled_pass,
"site_aware_pass": site_aware_pass,
}
class SegmentationGateTest(unittest.TestCase):
def test_expected_mask_scores(self):
self.assertEqual(dice(BASELINE, TRUTH), 0.75)
self.assertEqual(dice(TRUTH, TRUTH), 1.0)
self.assertEqual(dice(WORSE, TRUTH), 0.5)
def test_overall_gain_does_not_hide_site_regression(self):
result = evaluate(CASES)
self.assertEqual(result["overall_baseline"], 0.75)
self.assertEqual(result["overall_candidate"], 0.875)
self.assertEqual(result["overall_delta"], 0.125)
self.assertEqual(result["site_deltas"], {"A": 0.25, "B": -0.25})
self.assertTrue(result["pooled_pass"])
self.assertFalse(result["site_aware_pass"])
def test_missing_site_label_is_an_error(self):
broken = (replace(CASES[0], site=""),) + CASES[1:]
with self.assertRaisesRegex(ValueError, "site label"):
evaluate(broken)
if __name__ == "__main__":
unittest.main()
Run it locally or as a CI command:
python -m unittest -v test_segmentation_gate
The intended result is a passing test suite: the site-aware decision is expected to reject the candidate, so assertFalse is the successful assertion. This is a reference implementation with specified expected outcomes, not a reported test run.
To test your actual tooling, route these masks and site IDs through its scorer, aggregation code, and acceptance function. Keeping a separate implementation that your pipeline never calls would only test the example.
Keep the aggregation contract beside the gate
“Pooled Dice” is ambiguous unless the aggregation is specified. Averaging case-level Dice and calculating Dice after concatenating all pixels are different operations.
In this fixture, they produce the same overall value because every case has the same Dice denominator. Equal ground-truth size alone would not guarantee that equality if prediction sizes differed.
Use this short gate contract when adapting the fixture:
| Decision | Write down | Failure to catch |
|---|---|---|
| Scoring unit | Case, patient, volume, or another declared unit; include class and empty-mask policies | Slices or duplicate records silently changing weights |
| Aggregation | Case mean, site mean, or summed pixel counts; state the weighting | A refactor changing the meaning of the overall score |
| Membership | Expected case IDs and site IDs from a fixed evaluation manifest | Missing cases or an absent site disappearing from evaluation |
| Acceptance | Baseline identity, absolute tolerances, minimum support, and any required absolute score floors | An improved but still inadequate candidate passing a regression-only check |
A missing site must not become a passing site.
The example rejects blank labels and duplicate case IDs. It cannot detect that an entire site was omitted unless you also supply an expected manifest. In a pipeline, compare evaluated membership against that manifest before calculating acceptance, and surface a data-integrity error separately from a model regression.
Keep the baseline and candidate paired on the same cases. Otherwise, their difference mixes a prediction change with a dataset-composition change.
Use per-site rejection where the requirement is per-site
Keep the overall score as a summary, but require every declared site to meet its regression tolerance when that is the project's acceptance contract. Do not replace the case mean with an equal-weight site mean and assume the problem is solved: one site's gain can still offset another site's loss.
Choose real tolerances using the project's validation requirements and observed evaluation variability. A one-case site is useful for this deterministic software test, but it is insufficient evidence for a stable estimate of real site performance. If support is too low, report insufficient evidence under an explicit policy rather than silently treating it as a pass.
For a five-minute check today, save the fixture, run the assertions, and then connect it to your evaluator's public entry point. It verifies whether a site-specific failure survives aggregation; it does not establish clinical safety, generalization, or appropriate deployment thresholds.
For further work on reproducible pipeline checks, explore OpenThunder Dev.