perf: improve splitting of the offset vector in GroupColumn::take_n implementations, adding a new helper fn#23730
Open
mzabaluev wants to merge 3 commits into
Open
perf: improve splitting of the offset vector in GroupColumn::take_n implementations, adding a new helper fn#23730mzabaluev wants to merge 3 commits into
GroupColumn::take_n implementations, adding a new helper fn#23730mzabaluev wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23730 +/- ##
==========================================
+ Coverage 80.69% 80.70% +0.01%
==========================================
Files 1088 1089 +1
Lines 367766 368159 +393
Branches 367766 368159 +393
==========================================
+ Hits 296781 297135 +354
- Misses 53276 53316 +40
+ Partials 17709 17708 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
mzabaluev-flarion
force-pushed
the
take-n-offsets
branch
from
July 21, 2026 10:38
3c2e3ba to
69c2fda
Compare
A refinement of split_vec_min_alloc for splitting offset buffers. To improve performance of the current use in `BytesGroupValues::take_n`, this function adjusts the remaining offsets when they are copied to the new location, rather than two passes of copying then subtracting in place. It also accounts for n + 1 length of the resulting prefix vector, which current split_vec_min_alloc falls on the wrong side of.
mzabaluev-flarion
force-pushed
the
take-n-offsets
branch
from
July 21, 2026 10:40
69c2fda to
993d8b0
Compare
Benchmark the performance of take_n for ByteGroupValueBuilder in the context of the multi-group-by operation where it is used in production.
This is faster and could be reused with other GroupColumn implementations.
mzabaluev-flarion
force-pushed
the
take-n-offsets
branch
from
July 21, 2026 12:02
9bb1d54 to
e8a0f93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The
take_nimplementation forByteGroupValueBuilderusessplit_vec_min_allocto split the offset vector, then subtracts the n-th offset from the remaining offsets in place. This is slower than a single read-subtract-write pass.The implementation in #23648 will benefit from the same optimization.
What changes are included in this PR?
Add a utility function named
take_n_offsetsin datafusion-common, which minimizes allocation in the way ofsplit_vec_min_alloc, but also shifts the remaining offsets in a single pass with the copying.Are these changes tested?
Added a micro-benchmark that shows improvement on the
drain-like branch of the split, and no change on thesplit_off-like branch.To keep results in context, the timed function is
GroupValues::emiton a setup with two grouped columns, one of them of typeUtf8, the other an integer.Benchmark results (on a GCE instance type C4, Intel Xeon Emerald Rapids host CPU)
I set
--turbo-mode=ALL_CORE_MAXin the instance parameters and pinned CPU cores to reduce variation.Are there any user-facing changes?
utils::take_n_offsetsfunction is added in datafusion-common.