Skip to content

perf: optimize LEAD/LAG IGNORE NULLS evaluation#23711

Open
xudong963 wants to merge 1 commit into
apache:mainfrom
xudong963:perf-lead-lag-ignore-nulls
Open

perf: optimize LEAD/LAG IGNORE NULLS evaluation#23711
xudong963 wants to merge 1 commit into
apache:mainfrom
xudong963:perf-lead-lag-ignore-nulls

Conversation

@xudong963

@xudong963 xudong963 commented Jul 20, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

  • N/A

Rationale for this change

The whole-partition evaluation path for LEAD and LAG with IGNORE NULLS currently collects every valid row index, performs a binary search for every output row, converts every selected value to a ScalarValue, and finally rebuilds an Arrow array from those scalars.

This makes index selection O(n log m) for n rows and m non-null rows, and the per-row scalar materialization is particularly expensive for strings and nested values.

A local Criterion microbenchmark with 100,000 rows and 50% nulls measured the following speedups across the tested offsets:

Data type Speedup
Int64 6.7–6.9x
Utf8View 10.5–11.6x
List 21.7–21.8x

What changes are included in this PR?

  • Generate nullable Arrow gather indices with a single linear scan and bounded VecDeque state instead of collecting all valid indices and binary-searching for every row.
  • Materialize the result with Arrow's take kernel.
  • Use Arrow's zip kernel to fill non-null default values without constructing one ScalarValue per row.
  • Keep dictionary arrays with non-null defaults on scalar materialization because zip concatenates dictionaries and can overflow bounded dictionary key types.

The index generation and output materialization are now O(n). There are no public API changes.

Are these changes tested?

Yes.

Are there any user-facing changes?

No. This changes the implementation of whole-partition LEAD/LAG ... IGNORE NULLS evaluation without changing the SQL behavior or public API.

@github-actions github-actions Bot added the functions Changes to functions implementation label Jul 20, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.08333% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.68%. Comparing base (a8d1af6) to head (a3816b9).

Files with missing lines Patch % Lines
datafusion/functions-window/src/lead_lag.rs 52.08% 18 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23711      +/-   ##
==========================================
- Coverage   80.69%   80.68%   -0.01%     
==========================================
  Files        1088     1088              
  Lines      367741   367762      +21     
  Branches   367741   367762      +21     
==========================================
- Hits       296761   296745      -16     
- Misses      53272    53302      +30     
- Partials    17708    17715       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xudong963
xudong963 marked this pull request as ready for review July 20, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants