For example, give a decoder a 64-byte expanded-body cap and two concatenated gzip members that expand to 32 and 33 bytes. Each member fits individually, but their combined 65-byte body must be rejected before the handler runs.
The matching 64-byte body must reach the handler intact. A single gzip member containing 65 repeated A bytes must also be rejected, even though its compressed input fits below the cap. A check on the compressed request length cannot establish either behavior.
When a change adds gzip support or rewrites body handling, I would require an executable expanded-size check before accepting it.
Count across the whole decoded body
A compressed-input limit and an expanded-body limit protect different boundaries:
| Check | What it constrains | What it leaves unchecked |
|---|---|---|
| Request length header | The sender's declared encoded length | Actual expanded output |
| Counted compressed bytes | Encoded bytes accepted by the reader | Expansion during decompression |
| Cumulative decoded bytes | Output accepted from the decoder | CPU cost and total process memory |
A header cannot substitute for a decoded-byte counter, even when the header accurately describes the compressed body.
Two reset errors deserve explicit tests:
- Chunk reset: each decoded chunk fits below the cap, but their combined size does not.
- Member reset: each gzip member fits below the cap, but the concatenated members exceed it.
The limit belongs to the decoded body, not to each read or gzip member.
For a non-streaming application handler, I would buffer the accepted body within a configured cap and invoke the handler only after decoding completes. I would not use that design for bodies too large to buffer comfortably. Those need a different acceptance boundary, such as an isolated staging destination that can be discarded before application processing begins.
A runnable fixture with a one-byte overflow probe
Save this as test_gzip_limit.py. It uses only Python's standard library. The 64-byte cap makes the boundary cases easy to inspect; it is a test value, not a recommended deployment limit.
import gzip
import io
import unittest
from unittest.mock import Mock
class ExpandedBodyTooLarge(ValueError):
pass
def decode_gzip(encoded, limit, chunk_size=16 * 1024):
if limit < 0:
raise ValueError("limit must be nonnegative")
if chunk_size <= 0:
raise ValueError("chunk_size must be positive")
body = bytearray()
with gzip.GzipFile(fileobj=io.BytesIO(encoded), mode="rb") as decoded:
while True:
remaining = limit - len(body)
# Request only the allowance plus one overflow-detection byte.
piece = decoded.read(min(chunk_size, remaining + 1))
if not piece:
break
if len(piece) > remaining:
raise ExpandedBodyTooLarge(
f"expanded body exceeds {limit} bytes"
)
body.extend(piece)
return bytes(body)
def dispatch(encoded, handler, limit, chunk_size):
body = decode_gzip(encoded, limit, chunk_size)
return handler(body)
class GzipLimitTests(unittest.TestCase):
CAP = 64
@staticmethod
def member(data):
return gzip.compress(data, mtime=0)
def test_valid_bodies_reach_handler_intact(self):
cases = [
("empty", self.member(b""), b""),
("below", self.member(b"A" * 63), b"A" * 63),
("exact", self.member(b"A" * 64), b"A" * 64),
(
"members_exact",
self.member(b"A" * 32) + self.member(b"B" * 32),
b"A" * 32 + b"B" * 32,
),
]
for chunk_size in (1, 7, 64, 128):
for name, encoded, expected in cases:
with self.subTest(case=name, chunk_size=chunk_size):
handler = Mock(return_value="accepted")
result = dispatch(
encoded, handler, self.CAP, chunk_size
)
self.assertEqual(result, "accepted")
handler.assert_called_once_with(expected)
def test_oversized_bodies_never_reach_handler(self):
cases = [
("one_over", self.member(b"A" * 65)),
(
"members_one_over",
self.member(b"A" * 32) + self.member(b"B" * 33),
),
]
for chunk_size in (1, 7, 64, 128):
for name, encoded in cases:
with self.subTest(case=name, chunk_size=chunk_size):
# A compressed-size-only check would accept these.
self.assertLess(len(encoded), self.CAP)
handler = Mock()
with self.assertRaises(ExpandedBodyTooLarge):
dispatch(
encoded, handler, self.CAP, chunk_size
)
handler.assert_not_called()
if __name__ == "__main__":
unittest.main()
Run it locally or as a CI command:
python -m unittest -v test_gzip_limit.py
The expected result is acceptance of the valid cases and ExpandedBodyTooLarge for both oversized cases, with no handler call on rejection.
Why the extra byte matters
Suppose the decoder has already produced exactly 64 bytes. Stopping immediately would leave an unanswered question: is that the complete body, or is more output waiting?
The next read requests one byte. End-of-stream confirms that the body fits; another decoded byte proves overflow.
The overflow probe distinguishes an exact fit from an unread suffix.
The implementation keeps one counter through the entire operation: len(body). It never resets between reads. Python's gzip.GzipFile reads concatenated gzip members as a continuous decoded stream, so the same counter also spans member boundaries. That behavior is implemented in the member-handling loop in CPython's gzip reader.
The read request is bounded by both the configured chunk size and the remaining allowance plus one. No returned chunk is appended until it passes the size check. At most the configured limit is accumulated in body.
This explains the test choices:
| Boundary case | Expected behavior | Regression it exposes |
|---|---|---|
| 64 decoded bytes | Accept the complete body | Off-by-one rejection or truncation |
| 65 decoded bytes | Reject before handler invocation | Missing expanded-size enforcement |
| Small reads totaling 65 bytes | Reject | Counter reset between reads |
| Members totaling 32 + 33 bytes | Reject | Counter reset between members |
The concatenated exact-limit case matters too. Rejecting every multi-member input would pass the oversized-member test without implementing the acceptance behavior shown here. If your application deliberately forbids concatenated members, define and test that separate policy instead.
Put the assertion at the real application boundary
Adapt dispatch to call your actual body-ingestion path and keep the handler as a mock or isolated test sink. A helper-only test misses routes that bypass the helper. If middleware decompresses the request before your code sees it, place an integration check around that path so the test exercises the limit used by the deployed request flow.
Gzip limit review checklist
- Exact acceptance: the handler receives every byte of a valid body at the cap.
- Early rejection: an oversized body raises the intended error before handler invocation.
- Cumulative accounting: changing decoder read boundaries does not change acceptance.
- Member policy: concatenated members are either counted together or explicitly rejected.
The fixture's chunk_size controls decoded read sizes, not incoming HTTP transport chunks. Varying it tests accounting across decoder reads; it does not test how the HTTP ingestion path handles incoming chunks.
For an HTTP integration test, also assert the rejection response your application specifies. This fixture intentionally does not choose a status code or assume how a framework handles request decompression.
Keep the size guarantee narrow
The function bounds decoded bytes accumulated in its application buffer and bounds each requested read. It does not establish a total-memory or CPU bound.
The complete compressed input is already resident in this fixture. The gzip implementation has internal buffers, and converting the accumulated bytearray to bytes creates another allocation. A small returned read size is not a promise about every internal allocation.
Keep separate controls and tests for encoded input size, resource budgets, malformed or truncated streams, and your application's error handling. This fixture answers one precise question: can a decoded body exceed its cap and still reach the handler?
Port the exact-fit and concatenated one-byte-over cases into your real ingestion test today. Assert that the handler receives the accepted body intact and is never called on overflow.