Skip to content

gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig #112615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 4, 2023
Merged

gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig #112615

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
709466f
gh-105967: Work around a macOS bug, limit zlib crc32 calls to 1GiB.
gpshead Dec 2, 2023
f444127
Mention zipfile in the NEWS entry.
gpshead Dec 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2023-12-01-19-02-21.gh-issue-105967.Puq5Cn.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Workaround a bug in Apple's macOS platform zlib library where
:func:`zlib.crc32` and :func:`binascii.crc32` could produce incorrect results
on multi-gigabyte inputs. Including when using :mod:`zipfile` on zips
containing large data.
18 changes: 13 additions & 5 deletions Modules/binascii.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -770,12 +770,20 @@ binascii_crc32_impl(PyObject *module, Py_buffer *data, unsigned int crc)

Py_BEGIN_ALLOW_THREADS
/* Avoid truncation of length for very large buffers. crc32() takes
length as an unsigned int, which may be narrower than Py_ssize_t. */
while ((size_t)len > UINT_MAX) {
crc = crc32(crc, buf, UINT_MAX);
buf += (size_t) UINT_MAX;
len -= (size_t) UINT_MAX;
length as an unsigned int, which may be narrower than Py_ssize_t.
We further limit size due to bugs in Apple's macOS zlib.
See https://github.com/python/cpython/issues/105967
*/
#define ZLIB_CRC_CHUNK_SIZE 0x40000000
#if ZLIB_CRC_CHUNK_SIZE > INT_MAX
# error "unsupported less than 32-bit platform?"
#endif
while ((size_t)len > ZLIB_CRC_CHUNK_SIZE) {
crc = crc32(crc, buf, ZLIB_CRC_CHUNK_SIZE);
buf += (size_t) ZLIB_CRC_CHUNK_SIZE;
len -= (size_t) ZLIB_CRC_CHUNK_SIZE;
}
#undef ZLIB_CRC_CHUNK_SIZE
crc = crc32(crc, buf, (unsigned int)len);
Py_END_ALLOW_THREADS
} else {
Expand Down
18 changes: 13 additions & 5 deletions Modules/zlibmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1896,12 +1896,20 @@ zlib_crc32_impl(PyObject *module, Py_buffer *data, unsigned int value)

Py_BEGIN_ALLOW_THREADS
/* Avoid truncation of length for very large buffers. crc32() takes
length as an unsigned int, which may be narrower than Py_ssize_t. */
while ((size_t)len > UINT_MAX) {
value = crc32(value, buf, UINT_MAX);
buf += (size_t) UINT_MAX;
len -= (size_t) UINT_MAX;
length as an unsigned int, which may be narrower than Py_ssize_t.
We further limit size due to bugs in Apple's macOS zlib.
See https://github.com/python/cpython/issues/105967.
*/
#define ZLIB_CRC_CHUNK_SIZE 0x40000000
#if ZLIB_CRC_CHUNK_SIZE > INT_MAX
# error "unsupported less than 32-bit platform?"
#endif
while ((size_t)len > ZLIB_CRC_CHUNK_SIZE) {
value = crc32(value, buf, ZLIB_CRC_CHUNK_SIZE);
buf += (size_t) ZLIB_CRC_CHUNK_SIZE;
len -= (size_t) ZLIB_CRC_CHUNK_SIZE;
}
#undef ZLIB_CRC_CHUNK_SIZE
value = crc32(value, buf, (unsigned int)len);
Py_END_ALLOW_THREADS
} else {
Expand Down