Skip to content

FEAT Add safe_extract_zip helper for remote dataset loaders#1957

Draft
francose wants to merge 1 commit into
microsoft:mainfrom
francose:feat/safe-zip-extract
Draft

FEAT Add safe_extract_zip helper for remote dataset loaders#1957
francose wants to merge 1 commit into
microsoft:mainfrom
francose:feat/safe-zip-extract

Conversation

@francose

@francose francose commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Closes #1879.

so the gap is that the 3 remote dataset loaders (figstep, vlguard, jailbreakv-28k) all call zipfile.ZipFile.extractall() straight on whatever upstream zip they downloaded. extractall doesn't check member paths or sizes or entry types, so if any of those upstream zips ever get tampered with you get classic Zip Slip — single entry named ../../etc/cron.d/x and you've landed a cron job on whoever ran the loader. Symlink entries do the same thing without even needing ...

Adds a new helper pyrit/common/safe_extract.py and swaps the 3 call sites. The helper validates every member's metadata BEFORE writing anything to disk — if any member fails a check, nothing gets written. matters because otherwise a malicious zip with 99 valid entries + 1 bad one would leak 99 files before raising.

6 checks, each kills a different attack class:

  • realpath stays under destination → Zip Slip (../etc/x)
  • reject symlink / block / char / fifo / socket entries → symlink path escape
  • per-file size cap (default 1 GiB) → single-entry decompression DoS
  • total uncompressed size cap (default 5 GiB) → multi-file zip bomb
  • compression ratio cap (default 100×) → ratio bomb
  • file count cap (default 50,000) → inode / hash-table DoS

all caps are kwargs so individual callers can tighten them if they want.

note remote_dataset_loader.py also has a zipfile.ZipFile() block but it uses zf.open(inner) to stream a known-name file into memory, never writes by member name. different code path, not a Zip Slip site, left it alone.

16 tests in tests/unit/common/test_safe_extract.py covering happy path, dotdot traversal, absolute paths (unix and drive-letter), symlink / block / fifo entries, per-file bomb, total-size bomb, compression ratio bomb, file-count cap, no-partial-write guarantee, both bytes and path sources, dest auto-create, resolved-path return. all green locally.

on scope — Ruslan suggested in the thread that we drop Py 3.10/3.11 and switch to tar (PEP 706 has the data filter built in), which honestly would be cleaner if we could, but Roman flagged 3.11 has plenty of runway left and the upstream datasets aren't ours to convert anyway. app-level defensive extraction is the only fix that covers all supported runtimes today, which is what this does.

Signed-off-by: francose <13445813+francose@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Secure ZIP Extraction for Remote Datasets

1 participant