Skip to content

[core] Support TopN pushdown for BTree global indexes - #8991

Open
discivigour wants to merge 11 commits into
apache:masterfrom
discivigour:perf/btreeTopNPushdown
Open

[core] Support TopN pushdown for BTree global indexes#8991
discivigour wants to merge 11 commits into
apache:masterfrom
discivigour:perf/btreeTopNPushdown

Conversation

@discivigour

@discivigour discivigour commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add a TopN result type that keeps sortable keys and merges candidates globally across BTree readers
  • support forward and reverse BTree iteration for ascending and descending TopN, including null ordering
  • prune dominated BTree index files before lookup while retaining the complete file set for coverage calculation
  • group candidate row IDs by key and pass the resulting row ranges into data-evolution scans
  • add unit and table-level coverage for result merging, iterator direction, file selection, cross-file candidates, diagnostics, and fallback behavior

Why

BTree global indexes could evaluate predicates, but TopN planning could not use the index ordering to reduce index and data reads. Evaluating each file independently is insufficient because the final candidates must be compared globally, and coverage handling must not treat intentionally pruned index files as unindexed data.

This change computes a conservative global candidate set for supported single-column TopN queries. The execution engine still retains the final sort and limit, so the pushdown remains a pruning optimization rather than the source of final result ordering.

Impact

Data-evolution tables with a compatible BTree global index can reduce index-file opens, index-entry scans, and downstream row reads for ascending or descending TopN queries.

Validation

mvn -pl paimon-core -am \
  -Dmaven.repo.local=/opt/homebrew/opt/maven/repository \
  -DwildcardSuites=none \
  -DfailIfNoTests=false \
  -Dtest=GlobalIndexEvaluatorTest,TopNGlobalIndexResultTest,BTreeIndexReaderTest,LazyFilteredBTreeIndexReaderTest,BlockIteratorTest,BTreeTopNIndexFileSelectorTest,BtreeGlobalIndexTableTest \
  test
  • paimon-common: 229 tests passed
  • paimon-core: 27 tests passed
  • Checkstyle, Spotless, and Maven Enforcer passed
  • git diff --check passed

@leaves12138
leaves12138 dismissed their stale review August 3, 2026 02:40

Submitted by mistake.

@discivigour
discivigour marked this pull request as ready for review August 3, 2026 04:20

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add tests for Spark SQL?

@JingsongLi

Copy link
Copy Markdown
Contributor

Please disable global-index TopN for incremental scan domains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants