Test Log-to-Trace Correlation in CI with a Failing Fixture

AI-generated illustration. Choose coverage around the changed boundary: local capture checks identifier preservation, while backend ingestion and lookup require separate integration tests.

Test log-to-trace correlation in CI by capturing one synthetic failing request and requiring its error log to reference exactly one captured span through both trace_id and span_id. This local check verifies identifier preservation. Use a separate backend integration test to verify ingestion, indexing, or lookup.

In the supplied excerpt from 8 major updates to Cloudflare Observability, Cloudflare says it is launching eight updates bringing logs, traces, analytics, alerts, dashboards, querying, and telemetry export into one platform. That announcement leaves an application-side question: will a code change preserve the identifiers connecting an error log to its span? The excerpt does not establish specific correlation behavior, export protocols, or guarantees.

You can test identifier preservation without depending on a vendor. Start with a small contract validator and a broken fixture that shows it rejects a missing link.

Define the link before testing it

Use an application-owned schema rather than treating any vendor's payload as universal:

{
  "spans": [
    {"trace_id": "T", "span_id": "S"}
  ],
  "logs": [
    {
      "event": "synthetic.failure",
      "severity": "ERROR",
      "trace_id": "T",
      "span_id": "S"
    }
  ]
}

This exercise makes these assumptions:

• The capture belongs to one isolated synthetic request.
• The request produces one completed span and one target error log.
• The error is logged while the request span is active. There are no child spans.
• Log and span capture are enabled, with no sampling drops for this test.
• T and S are opaque fixture identifiers, not examples of a telemetry protocol's valid wire format.

The contract is: exactly one target error log must contain both identifiers, and exactly one captured span must match the pair.

Checking only trace_id cannot establish a span-level link because multiple spans can belong to the same trace. Field presence alone is insufficient, too: both identifiers could exist without matching any captured span.

If your application deliberately logs inside a child span, change the expected relationship. Do not require the root request span just because this example has only one span.

Make the missing field fail explicitly

Save this as verify_telemetry.py. It uses only Python's standard library. The schema is intentionally narrow, not a complete telemetry-format validator.

import argparse
import copy
import json
from pathlib import Path


class ContractError(ValueError):
    pass


def require(condition, message):
    if not condition:
        raise ContractError(message)


def validate(bundle):
    require(isinstance(bundle, dict), "capture must be an object")
    for key in ("spans", "logs"):
        require(isinstance(bundle.get(key), list), f"{key} must be an array")
        require(
            all(isinstance(item, dict) for item in bundle[key]),
            f"{key} entries must be objects",
        )

    errors = [
        log for log in bundle["logs"]
        if log.get("event") == "synthetic.failure"
        and log.get("severity") == "ERROR"
    ]
    require(
        len(errors) == 1,
        f"expected one synthetic error log, got {len(errors)}",
    )
    error = errors[0]

    for field in ("trace_id", "span_id"):
        value = error.get(field)
        require(
            isinstance(value, str) and bool(value.strip()),
            f"error log missing {field}",
        )

    matches = [
        span for span in bundle["spans"]
        if span.get("trace_id") == error["trace_id"]
        and span.get("span_id") == error["span_id"]
    ]
    require(
        len(matches) == 1,
        f"expected one matching captured span, got {len(matches)}",
    )


def fixture_checks():
    good = {
        "spans": [{"trace_id": "T", "span_id": "S"}],
        "logs": [{
            "event": "synthetic.failure",
            "severity": "ERROR",
            "trace_id": "T",
            "span_id": "S",
        }],
    }
    validate(good)

    broken = copy.deepcopy(good)
    del broken["logs"][0]["span_id"]
    try:
        validate(broken)
    except ContractError as exc:
        require(str(exc) == "error log missing span_id", str(exc))
    else:
        raise ContractError("broken fixture unexpectedly passed")

    broken["logs"][0]["span_id"] = "S"
    validate(broken)
    print("fixture expectations satisfied")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("capture", nargs="?")
    parser.add_argument("--self-test", action="store_true")
    args = parser.parse_args()
    if not args.self_test and not args.capture:
        parser.error("provide a capture file or --self-test")
    try:
        if args.self_test:
            fixture_checks()
        else:
            validate(json.loads(Path(args.capture).read_text()))
            print("telemetry contract satisfied")
    except ContractError as exc:
        raise SystemExit(str(exc))

Run the fixture exercise:

python verify_telemetry.py --self-test

These are expected results, not reported test runs:

The self-test catches the intentional failure, so its overall expected exit status is zero. Validating a capture file with that missing field instead produces the diagnostic and a nonzero exit status.

The target-log count matters. Without it, a validator that loops over error logs could silently pass when the logging path emits nothing.

AI-generated illustration. The broken fixture removes spanid from the error log. A trace identifier alone is insufficient: both identifiers must match exactly one captured span.

Feed the validator captured application output

Fixtures test the validator. Captured request output tests the instrumentation path that request exercises.

Save the following isolated demo as capture_request.py. It dispatches an in-process synthetic request, logs a deliberately raised exception while a span context is active, and records the span when the request completes. It uses no network service, credentials, or customer data.

This is hand-written demonstration instrumentation with a synchronous capture sink, not a tracing protocol implementation or vendor SDK exporter.

import json
import logging
import secrets
from contextvars import ContextVar


capture = {"spans": [], "logs": []}
active_span = ContextVar("active_span", default=None)


class CaptureHandler(logging.Handler):
    def emit(self, record):
        identifiers = dict(active_span.get() or {})
        capture["logs"].append({
            "event": record.getMessage(),
            "severity": record.levelname,
            **identifiers,
        })


logger = logging.getLogger("telemetry-contract-demo")
logger.setLevel(logging.ERROR)
logger.propagate = False
logger.addHandler(CaptureHandler())


class SyntheticFailure(Exception):
    pass


def request():
    identifiers = {
        "trace_id": secrets.token_hex(16),
        "span_id": secrets.token_hex(8),
    }
    token = active_span.set(identifiers)
    try:
        try:
            raise SyntheticFailure("isolated test failure")
        except SyntheticFailure:
            logger.error("synthetic.failure")
            return 500
    finally:
        # Record the completed span before releasing its context.
        capture["spans"].append(dict(identifiers))
        active_span.reset(token)


if __name__ == "__main__":
    status = request()
    if status != 500:
        raise SystemExit(f"expected synthetic status 500, got {status}")
    print(json.dumps(capture))

Wire the captured output to the same validator:

set -eu
python verify_telemetry.py --self-test
python capture_request.py > captured.json
python verify_telemetry.py captured.json

The expected result is a successful contract check: the logging handler reads the active identifiers, and the completed span carries the same pair. The command sequence stops if either script fails.

This demo does not verify your existing service. To make it an application regression test, keep validate() and replace the demo request and capture sink with your application's real request path and locally captured telemetry.

Capture at the boundary you need to protect

For an application check:

1. Create a fresh capture destination.
2. Start the application with its real instrumentation and a supported local capture mechanism.
3. Send one synthetic request to a test-only failure path, using an isolated stub dependency if needed.
4. Verify that the request reached the intended failure path.
5. Wait for request and span completion, then flush the relevant telemetry pipelines using their documented mechanisms.
6. Normalize the captured records into the application-owned schema and call validate().

Choose the capture boundary deliberately. Capturing records before a serialization or export adapter cannot detect that adapter dropping span_id. Capturing its output can, provided the harness exercises it.

The normalizer may rename fields, map severity representations, or convert identifier encodings consistently. It must not manufacture a missing log identifier from request state or copy it from a captured span. That would hide the defect.

Keep the raw capture as a CI artifact alongside the normalized document. Together, they help distinguish missing application context from a normalization mistake.

Make the request lifecycle deterministic

A missing matching span can indicate a broken link or an incomplete capture. Before treating the failure as an instrumentation regression, check:

• Request completion: Confirm that the expected synthetic response or exception handling path finished. An early connection failure is not the intended error-log test.
• Span completion: Wait for the span under test to end. For background work, wait for an explicit completion signal rather than assuming response completion is enough.
• Exporter flushing: Drain both log and span pipelines before reading the capture. Treat flush failures and timeouts as capture failures; do not replace synchronization with an arbitrary sleep.
• Sampling and filtering: Configure the isolated test to retain its span and error log. This checks instrumentation with retention enabled, not the production sampling policy.
• Capture isolation: Use a fresh per-test sink or isolated process. Shared buffers can introduce records from other tests and make matching ambiguous.
• Normalization fidelity: Preserve absent fields as absent. Verify that identifier conversions preserve equality without inventing values.
• Async boundaries: If the regression risk involves a task, thread, or queue boundary, make the synthetic path cross it. The synchronous demo cannot prove propagation across that boundary.

In this exercise, zero matching spans and duplicate matching spans both fail. If your real capture can include delivery retries, define that behavior separately. Do not silently deduplicate records to make the contract pass.

Decide whether a backend test is also necessary

The local check supports a narrow claim: in this captured request, the target error log preserves a link to exactly one captured span. It does not prove that all request paths are instrumented or that production telemetry will always be retained.

Use the changed boundary to decide what else to test:

A backend test should emit a uniquely identifiable synthetic request into an isolated test destination, wait for visibility with bounded polling, and execute the supported lookup that matters to your workflow. Finding raw records does not prove that a dashboard link works. Nor does a passing local contract establish ingestion, indexing, query behavior, or rendering.

Keep the local validator even when you add backend coverage. Its broken fixture gives you a specific diagnostic for identifier loss without mixing that failure with credentials, network availability, or backend visibility delays.

For a next step, explore OpenThunder as you plan where this contract belongs in your CI verification gates.