Reading gzip headers can lead to OOM for adversarial inputs

[KowalskiThomas]

Bug report

Bug description:

In short

_read_until_null (added a few days ago in dd94457), appends every byte from a stream into a bytearray with no upper bound (here). Any gzip stream that sets FNAME or FCOMMENT together with any other flag (bypassing the early-exit at the flag == FNAME fast-path) and never emits a NUL byte will cause the header bytearray to grow until memory is exhausted. This occurs before any decompression takes place, so existing mitigations for zip-bomb-like inputs that bound decompressed output size do not apply.

Proposed fix

Cap _read_until_null at a reasonable per-field limit and raise BadGzipFile if exceeded:

_MAX_GZIP_HEADER_FIELD_SIZE = 65536  # consistent with max FEXTRA length (16-bit)

def _read_until_null(fp, append_to: bytearray) -> None:
    for _ in range(_MAX_GZIP_HEADER_FIELD_SIZE + 1):
        s = fp.read(1)
        append_to += s
        if not s or s == b'\000':
            return
    raise BadGzipFile('Header field exceeds maximum size '
                      f'({_MAX_GZIP_HEADER_FIELD_SIZE} bytes)')

(65535 bytes was picked to match the maximum FEXTRA field size (a 16-bit length prefix), making all variable-length header fields consistent.)

I have a branch that adds a mitigation here in my fork.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-149945: Fix potential OOM for gzip with large header #149979

[serhiy-storchaka]

Good catch @KowalskiThomas.

I do not think that there is a need to impose an artificial limit on the size of the filename and comment. We can update the CRC incrementally, without reading the whole header in memory.

[serhiy-storchaka]

Note that for triggering this issue you need to read a gzip file with the size of many hundreds of megabytes or even gigabytes. On my computer, reading the header of a gzip file with a gigabyte long filename or comment takes about 2.5 minutes without the CRC calculation (and twice as much with it). So this is an issue even without CRC. We may impose some limits, but this is a different issue.

[KowalskiThomas]

Ah, I had a reproducer script that I forgot to include (which allowed me to trigger the bug more easily by generating bytes in memory instead of reading an actual file), but yeah you do need a somewhat big payload to begin with.