IO: Fix Windows drive letters misidentified as URI schemes - #3721
Conversation
On Windows, Python's urlparse treats paths like 'C:\Users\...' as having scheme='c', which causes 'Unrecognized filesystem type in URI: c' errors. This adds a platform-guarded predicate that detects single-character alphabetic schemes on Windows and remaps them to 'file', enabling correct local filesystem routing. Fixes all three parse sites (_infer_file_io_from_scheme, PyArrowFileIO.parse_location, FsspecFileIO._get_fs_from_uri) and includes platform-conditional tests. Related: apache#2477, apache#1005
| scheme='c'. This detects that case so callers can route to the local filesystem. | ||
| Only returns True on Windows — no behavior change on other platforms. | ||
| """ | ||
| return sys.platform == "win32" and len(scheme) == 1 and scheme.isalpha() |
There was a problem hiding this comment.
Windows newb here:
I'm looking at https://stackoverflow.com/questions/446209/possible-values-from-sys-platform. There's a couple other Windows derivatives mentioned. Should we do that for these too?
There was a problem hiding this comment.
Good question! My understanding is that "win32" is sufficient here, the drive letter parsing bug only exists on native Windows where paths look like C:\....
The other sys.platform values from that SO thread don't need coverage because they use POSIX paths, so urlparse never produces a single-letter scheme on them:
- Cygwin (
sys.platform == "cygwin") translatesC:\Users\...to/cygdrive/c/Users/...at the filesystem layer (Cygwin docs) - MSYS2 (
sys.platform == "msys") similarly mapsC:\Usersto/c/Users(MSYS2 docs)
sys.platform == "win32" is also the standard CPython idiom for this, it's how the stdlib distinguishes Windows from POSIX behavior (sys.platform docs), and what pytest recommends for skipif markers.
Rationale
On Windows,
urlparse('C:\Users\...')producesscheme='c', causing:This breaks all local file operations for Windows users.
The Fix
A single platform-guarded predicate shared across all three affected parse sites:
When detected, the scheme is remapped to
'file', which routes to PyArrow/fsspec's local filesystem which already handles Windows paths natively.Changes
pyiceberg/io/__init__.py_infer_file_io_from_schemepyiceberg/io/pyarrow.pyparse_locationpyiceberg/io/fsspec.py_get_fs_from_uritests/io/test_io.pytests/io/test_pyarrow.pyparse_locationWindows testWhy all three sites?
They execute at different lifecycle stages and are reachable through independent code paths. Fixing only one (as PR #3161 did) leaves the others broken.
Regression safety
The
sys.platform == 'win32'guard means this is literally invisible on Linux/macOS the predicate short-circuits toFalsewithout inspecting the scheme. Existing CI onubuntu-latestcannot observe any change.Are these changes tested?
Yes. 11 new test cases covering:
@pytest.mark.skipifAll existing tests continue to pass.
Closes
Closes #3720
Related: #2477, #1005