Skip to content

IO: Extract PyArrowFileIO into dedicated module - #3738

Open
qzyu999 wants to merge 1 commit into
apache:mainfrom
qzyu999:extract-pyarrow-fileio
Open

IO: Extract PyArrowFileIO into dedicated module#3738
qzyu999 wants to merge 1 commit into
apache:mainfrom
qzyu999:extract-pyarrow-fileio

Conversation

@qzyu999

@qzyu999 qzyu999 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Extract PyArrowFile, PyArrowFileIO, PyArrowLocalFileSystem and their helper functions into a new pyiceberg/io/_pyarrow_fileio.py module. The original pyiceberg/io/pyarrow.py re-exports all public names for full backward compatibility.

This is the first step toward decomposing the 3,100+ line pyarrow.py monolith into focused modules by concern (tracked in #3737).

Changes

  • New file: pyiceberg/io/_pyarrow_fileio.py (584 lines) containing the FileIO implementation
  • Modified: pyiceberg/io/pyarrow.py (-537 lines) replaced inline definitions with re-exports
  • Removed unused imports from pyarrow.py that were only needed by the FileIO section

What stays the same

  • All existing from pyiceberg.io.pyarrow import PyArrowFileIO imports continue to work
  • Zero behavior change pure code movement with re-exports
  • Pickle/unpickle of PyArrowFileIO works unchanged
  • No new dependencies

Why

pyarrow.py handles 6 unrelated concerns in one file. The FileIO concern (filesystem abstraction for S3/GCS/Azure/HDFS/local) has zero coupling to schema conversion, expression translation, or statistics collection. Extracting it:

  1. Reduces cognitive load when working on scan/write logic
  2. Establishes the pattern for subsequent extractions (schema, expressions, stats, write, scan)
  3. Creates a clean seam point for future alternative FileIO implementations

Testing

Full unit test suite passes: 3803 passed, 3 skipped, 0 failures, 0 errors

(Windows local testing requires #3721 + #3722 applied on top; Linux CI should pass without them.)

Move PyArrowFile, PyArrowFileIO, PyArrowLocalFileSystem and their helpers (_cached_resolve_s3_region, _import_retry_strategy) into pyiceberg/io/_pyarrow_fileio.py. The original pyiceberg/io/pyarrow.py re-exports all public names for full backward compatibility.

This is the first step toward decomposing the 3100+ line pyarrow.py monolith into focused modules by concern, enabling future pluggable compute engines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant