Skip to content

test: add partitioned schema evolution coverage#406

Open
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:test/258-partitioned-schema-evolution
Open

test: add partitioned schema evolution coverage#406
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:test/258-partitioned-schema-evolution

Conversation

@liujiwen-up

@liujiwen-up liujiwen-up commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: #258

Add integration coverage for partitioned schema evolution across mixed file formats. The fixture writes old-schema partitioned Parquet data, adds an extra column, then writes new-schema ORC and Avro data across multiple dt partitions.

Brief change log

  • Add partitioned_format_schema_evolution_add_column Spark fixture with dt STRING partitioning.
  • Add core read coverage for partition pruning, projection plus filter on partition/data columns, and extra IS NULL with a partition filter.
  • Add DataFusion SQL coverage for partition filters, added-column null fill, and projected partition/data columns.

Tests

  • python3 -m py_compile dev/spark/provision.py: passed
  • cargo fmt --check: passed
  • cargo test -p paimon-integration-tests --test read_tables partitioned_format_schema_evolution -- --list: passed
  • cargo test -p paimon-datafusion --test read_tables partitioned_format_schema_evolution -- --list: passed
  • cargo test -p paimon-integration-tests --test read_tables partitioned_format_schema_evolution --no-run: passed
  • cargo test -p paimon-datafusion --test read_tables partitioned_format_schema_evolution --no-run: passed
  • cargo test -p paimon-integration-tests --test read_tables partitioned_format_schema_evolution -- --nocapture: passed after re-provisioning the Spark warehouse
  • cargo test -p paimon-datafusion --test read_tables partitioned_format_schema_evolution -- --nocapture: passed after re-provisioning the Spark warehouse

API and Format

No API or storage format changes. This PR only adds Spark fixture data and integration test coverage.

Documentation

No documentation update required; this is test-only coverage.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants