Skip to content

Commit 9f8741b

Browse files
committed
Skip Puffin delete files (v3 delete vectors) in positional delete handling
Iceberg v3 uses Puffin files for delete vectors (DVs), which have a different format than Parquet position delete files. The current implementation only supports Parquet position delete files. Changes: - Filter position delete files to only include Parquet format - Emit a warning when unsupported delete file formats (e.g., Puffin DVs) are skipped - Results may include rows that should have been deleted when DVs are present This is a temporary limitation until Puffin DV support is implemented.
1 parent d5830a4 commit 9f8741b

1 file changed

Lines changed: 27 additions & 2 deletions

File tree

pyiceberg/execution/_orchestrate.py

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
from typing import TYPE_CHECKING, Any, Literal, TypeVar
3838

3939
from pyiceberg.expressions import AlwaysTrue
40-
from pyiceberg.manifest import DataFileContent
40+
from pyiceberg.manifest import DataFileContent, FileFormat
4141
from pyiceberg.schema import Schema
4242
from pyiceberg.table.sorting import UNSORTED_SORT_ORDER_ID
4343

@@ -178,7 +178,32 @@ def orchestrate_scan(
178178
def _execute_task(task: FileScanTask) -> list[pa.RecordBatch]:
179179
"""Execute a single scan task: read, resolve deletes, filter, reconcile schema."""
180180
eq_deletes = [d for d in task.delete_files if d.content == DataFileContent.EQUALITY_DELETES]
181-
pos_deletes = [d for d in task.delete_files if d.content == DataFileContent.POSITION_DELETES]
181+
# Only include Parquet position delete files. Puffin files (delete vectors in v3)
182+
# require different handling that's not yet implemented - they will be silently
183+
# skipped, which may return a superset of correct results.
184+
pos_deletes = [
185+
d
186+
for d in task.delete_files
187+
if d.content == DataFileContent.POSITION_DELETES and d.file_format == FileFormat.PARQUET
188+
]
189+
190+
# Warn if there are unsupported delete file formats (e.g., Puffin DVs)
191+
unsupported_deletes = [
192+
d
193+
for d in task.delete_files
194+
if d.content == DataFileContent.POSITION_DELETES and d.file_format != FileFormat.PARQUET
195+
]
196+
if unsupported_deletes:
197+
import warnings
198+
199+
formats = {d.file_format.name for d in unsupported_deletes}
200+
warnings.warn(
201+
f"Skipping {len(unsupported_deletes)} position delete file(s) with unsupported format(s): {formats}. "
202+
f"Delete vectors (Puffin files) in Iceberg v3 are not yet supported. "
203+
f"Results may include rows that should have been deleted.",
204+
UserWarning,
205+
stacklevel=2,
206+
)
182207

183208
if pos_deletes and eq_deletes:
184209
batches: Iterator[pa.RecordBatch] = _apply_positional_deletes(

0 commit comments

Comments
 (0)