Skip to content

Commit 049c412

Browse files
bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files (GH-25058)
`shutil.unpack_archive()` tries to read the whole file into memory, making no use of any kind of smaller buffer. Process crashes for really large files: I.e. archive: ~1.7G, unpacked: ~10G. Before the crash it can easily take away all available RAM on smaller systems. Had to pull the code form `zipfile.Zipfile.extractall()` to fix this Automerge-Triggered-By: GH:gpshead (cherry picked from commit f32c795) Co-authored-by: Igor Bolshakov <[email protected]>
1 parent 2057ce8 commit 049c412

File tree

2 files changed

+8
-10
lines changed

2 files changed

+8
-10
lines changed

Lib/shutil.py

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1159,20 +1159,16 @@ def _unpack_zipfile(filename, extract_dir):
11591159
if name.startswith('/') or '..' in name:
11601160
continue
11611161

1162-
target = os.path.join(extract_dir, *name.split('/'))
1163-
if not target:
1162+
targetpath = os.path.join(extract_dir, *name.split('/'))
1163+
if not targetpath:
11641164
continue
11651165

1166-
_ensure_directory(target)
1166+
_ensure_directory(targetpath)
11671167
if not name.endswith('/'):
11681168
# file
1169-
data = zip.read(info.filename)
1170-
f = open(target, 'wb')
1171-
try:
1172-
f.write(data)
1173-
finally:
1174-
f.close()
1175-
del data
1169+
with zip.open(name, 'r') as source, \
1170+
open(targetpath, 'wb') as target:
1171+
copyfileobj(source, target)
11761172
finally:
11771173
zip.close()
11781174

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Fix :exc:`MemoryError` in :func:`shutil.unpack_archive` which fails inside
2+
:func:`shutil._unpack_zipfile` on large files. Patch by Igor Bolshakov.

0 commit comments

Comments
 (0)