feat(trace-utils)!: add dedup convenience to VecMap by yannham · Pull Request #2049 · DataDog/libdatadog

yannham · 2026-05-28T09:15:53Z

What does this PR do?

Follow-up of #2022. This split off some work required in #2043 for deduplication before msgpack encoding as a separate PR.

Motivation

See #2022 for overall motivation. To integrate VecMap, it turns out we can't currently rely on the agent to properly handle maps with deuplicate entries (the "last win" semantics is the current implementation, but not guaranteed). We need to perform deduplication before msgpack encoding. This PR backports the required machinery into VecMap as a separate PR.

Additional Notes

N/A

How to test the change?

Tests included.

github-actions · 2026-05-28T09:17:45Z

📚 Documentation Check Results

⚠️ 602 documentation warning(s) found

📦 `libdd-trace-utils` - 602 warning(s)

Updated: 2026-06-01 17:20:30 UTC | Commit: 9fee767 | missing-docs job results

github-actions · 2026-05-28T09:18:30Z

Clippy Allow Annotation Report

Comparing clippy allow annotations between branches:

Base Branch: origin/main
PR Branch: origin/yannham/vecmap-as-dedup

Summary by Rule

Rule	Base Branch	PR Branch	Change

Annotation Counts by File

File	Base Branch	PR Branch	Change

Annotation Stats by Crate

Crate	Base Branch	PR Branch	Change
`clippy-annotation-reporter`	5	5	No change (0%)
`datadog-ffe-ffi`	1	1	No change (0%)
`datadog-ipc`	21	21	No change (0%)
`datadog-live-debugger`	6	6	No change (0%)
`datadog-live-debugger-ffi`	10	10	No change (0%)
`datadog-profiling-replayer`	4	4	No change (0%)
`datadog-remote-config`	3	3	No change (0%)
`datadog-sidecar`	57	57	No change (0%)
`libdd-common`	13	13	No change (0%)
`libdd-common-ffi`	12	12	No change (0%)
`libdd-data-pipeline`	5	5	No change (0%)
`libdd-ddsketch`	2	2	No change (0%)
`libdd-dogstatsd-client`	1	1	No change (0%)
`libdd-profiling`	13	13	No change (0%)
`libdd-telemetry`	20	20	No change (0%)
`libdd-tinybytes`	4	4	No change (0%)
`libdd-trace-normalization`	2	2	No change (0%)
`libdd-trace-obfuscation`	3	3	No change (0%)
`libdd-trace-stats`	1	1	No change (0%)
`libdd-trace-utils`	13	13	No change (0%)
Total	196	196	No change (0%)

About This Report

This report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality.

github-actions · 2026-05-28T09:19:47Z

🔒 Cargo Deny Results

⚠️ 4 issue(s) found, showing only errors (advisories, bans, sources)

📦 `libdd-trace-utils` - 4 error(s)

Show output

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:177:1
    │
177 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by @lopopolo) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── (dev) libdd-common v4.2.0
      │   ├── libdd-capabilities-impl v2.0.0
      │   │   └── libdd-trace-utils v6.0.1
      │   │       └── (dev) libdd-trace-utils v6.0.1 (*)
      │   └── libdd-trace-utils v6.0.1 (*)
      ├── (dev) libdd-trace-normalization v2.0.0
      │   └── libdd-trace-utils v6.0.1 (*)
      ├── libdd-trace-utils v6.0.1 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v6.0.1 (*)

error[vulnerability]: Name constraints for URI names were incorrectly accepted
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:199:1
    │
199 │ rustls-webpki 0.103.10 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0098
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0098
    ├ Name constraints for URI names were ignored and therefore accepted.
      
      Note this library does not provide an API for asserting URI names, and URI name constraints are otherwise not implemented.  URI name constraints are now rejected unconditionally.
      
      Since name constraints are restrictions on otherwise properly-issued certificates, this bug is reachable only after signature verification and requires misissuance to exploit.
      
      This vulnerability is identified as [GHSA-965h-392x-2mh5](https://github.com/rustls/webpki/security/advisories/GHSA-965h-392x-2mh5). Thank you to @1seal for the report.
    ├ Solution: Upgrade to >=0.103.12, <0.104.0-alpha.1 OR >=0.104.0-alpha.6 (try `cargo update -p rustls-webpki`)
    ├ rustls-webpki v0.103.10
      └── rustls v0.23.37
          ├── hyper-rustls v0.27.7
          │   └── libdd-common v4.2.0
          │       ├── libdd-capabilities-impl v2.0.0
          │       │   └── libdd-trace-utils v6.0.1
          │       │       └── (dev) libdd-trace-utils v6.0.1 (*)
          │       └── libdd-trace-utils v6.0.1 (*)
          ├── libdd-common v4.2.0 (*)
          └── tokio-rustls v0.26.0
              ├── hyper-rustls v0.27.7 (*)
              └── libdd-common v4.2.0 (*)

error[vulnerability]: Name constraints were accepted for certificates asserting a wildcard name
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:199:1
    │
199 │ rustls-webpki 0.103.10 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0099
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0099
    ├ Permitted subtree name constraints for DNS names were accepted for certificates asserting a wildcard name.
      
      This was incorrect because, given a name constraint of `accept.example.com`, `*.example.com` could feasibly allow a name of `reject.example.com` which is outside the constraint.
      This is very similar to [CVE-2025-61727](https://go.dev/issue/76442).
      
      Since name constraints are restrictions on otherwise properly-issued certificates, this bug is reachable only after signature verification and requires misissuance to exploit.
      
      This vulnerability is identified as [GHSA-xgp8-3hg3-c2mh](https://github.com/rustls/webpki/security/advisories/GHSA-xgp8-3hg3-c2mh). Thank you to @1seal for the report.
    ├ Solution: Upgrade to >=0.103.12, <0.104.0-alpha.1 OR >=0.104.0-alpha.6 (try `cargo update -p rustls-webpki`)
    ├ rustls-webpki v0.103.10
      └── rustls v0.23.37
          ├── hyper-rustls v0.27.7
          │   └── libdd-common v4.2.0
          │       ├── libdd-capabilities-impl v2.0.0
          │       │   └── libdd-trace-utils v6.0.1
          │       │       └── (dev) libdd-trace-utils v6.0.1 (*)
          │       └── libdd-trace-utils v6.0.1 (*)
          ├── libdd-common v4.2.0 (*)
          └── tokio-rustls v0.26.0
              ├── hyper-rustls v0.27.7 (*)
              └── libdd-common v4.2.0 (*)

error[vulnerability]: Reachable panic in certificate revocation list parsing
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:199:1
    │
199 │ rustls-webpki 0.103.10 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0104
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0104
    ├ A panic was reachable when parsing certificate revocation lists via [`BorrowedCertRevocationList::from_der`]
      or [`OwnedCertRevocationList::from_der`].  This was the result of mishandling a syntactically valid empty
      `BIT STRING` appearing in the `onlySomeReasons` element of a `IssuingDistributionPoint` CRL extension.
      
      This panic is reachable prior to a CRL's signature being verified.
      
      Applications that do not use CRLs are not affected.
      
      Thank you to @tynus3 for the report.
    ├ Solution: Upgrade to >=0.103.13, <0.104.0-alpha.1 OR >=0.104.0-alpha.7 (try `cargo update -p rustls-webpki`)
    ├ rustls-webpki v0.103.10
      └── rustls v0.23.37
          ├── hyper-rustls v0.27.7
          │   └── libdd-common v4.2.0
          │       ├── libdd-capabilities-impl v2.0.0
          │       │   └── libdd-trace-utils v6.0.1
          │       │       └── (dev) libdd-trace-utils v6.0.1 (*)
          │       └── libdd-trace-utils v6.0.1 (*)
          ├── libdd-common v4.2.0 (*)
          └── tokio-rustls v0.26.0
              ├── hyper-rustls v0.27.7 (*)
              └── libdd-common v4.2.0 (*)

advisories FAILED, bans ok, sources ok

Updated: 2026-06-01 17:22:12 UTC | Commit: 9fee767 | dependency-check job results

datadog-datadog-prod-us1-2 · 2026-05-28T09:21:57Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 88.04%
• Overall Coverage: 73.07% (+0.17%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0996e9a | Docs | Datadog PR Page | Give us feedback!}

codecov-commenter · 2026-05-28T13:28:11Z

Codecov Report

❌ Patch coverage is 88.04348% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.07%. Comparing base (f7d471d) to head (0996e9a).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2049      +/-   ##
==========================================
+ Coverage   72.90%   73.07%   +0.16%     
==========================================
  Files         460      460              
  Lines       76396    76789     +393     
==========================================
+ Hits        55696    56110     +414     
+ Misses      20700    20679      -21

Components	Coverage Δ
libdd-crashtracker	`65.47% <ø> (+0.24%)`	⬆️
libdd-crashtracker-ffi	`37.68% <ø> (+0.85%)`	⬆️
libdd-alloc	`98.77% <ø> (ø)`
libdd-data-pipeline	`85.84% <ø> (+0.23%)`	⬆️
libdd-data-pipeline-ffi	`77.03% <ø> (+1.33%)`	⬆️
libdd-common	`79.89% <ø> (ø)`
libdd-common-ffi	`74.41% <ø> (ø)`
libdd-telemetry	`73.34% <ø> (ø)`
libdd-telemetry-ffi	`31.36% <ø> (ø)`
libdd-dogstatsd-client	`82.64% <ø> (ø)`
datadog-ipc	`74.75% <ø> (-1.47%)`	⬇️
libdd-profiling	`81.69% <ø> (ø)`
libdd-profiling-ffi	`64.79% <ø> (ø)`
libdd-sampling	`97.41% <ø> (-0.05%)`	⬇️
datadog-sidecar	`29.37% <ø> (+0.17%)`	⬆️
datdog-sidecar-ffi	`11.06% <ø> (+0.88%)`	⬆️
spawn-worker	`48.86% <ø> (ø)`
libdd-tinybytes	`93.80% <ø> (ø)`
libdd-trace-normalization	`81.71% <ø> (ø)`
libdd-trace-obfuscation	`87.30% <ø> (ø)`
libdd-trace-protobuf	`68.25% <ø> (ø)`
libdd-trace-utils	`89.19% <88.04%> (+0.25%)`	⬆️
libdd-tracer-flare	`86.88% <ø> (ø)`
libdd-log	`74.83% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dd-octo-sts · 2026-05-28T13:49:22Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	7.63 MB	7.63 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	82.94 MB	82.94 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	94.02 MB	94.02 MB	0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.25 MB	10.25 MB	0% (0 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	24.54 MB	24.54 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	83.96 KB	83.96 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	178.23 MB	178.26 MB	+.01% (+24.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	915.25 MB	915.26 MB	+0% (+17.22 KB) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.03 MB	8.03 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	83.96 KB	83.96 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	23.77 MB	23.77 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	47.43 MB	47.43 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	21.27 MB	21.27 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	85.29 KB	85.29 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	182.23 MB	182.23 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	908.37 MB	908.39 MB	+0% (+16.97 KB) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.20 MB	6.20 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	85.29 KB	85.29 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	25.48 MB	25.48 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	45.07 MB	45.07 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	73.95 MB	73.95 MB	0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	8.52 MB	8.52 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	89.36 MB	89.36 MB	0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.36 MB	10.36 MB	0% (0 B) 👌

Avoid the redundant collect from HashMap back into Vec when deduping. The Owned variant now holds a HashMap<&K, &V> directly, removing one allocation and copy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat(native-span): add dedup convenience to VecMap

9bdd9ed

yannham requested a review from paullegranddc May 28, 2026 09:15

yannham requested review from a team as code owners May 28, 2026 09:15

github-actions Bot added the mini-agent label May 28, 2026

paullegranddc reviewed May 28, 2026

View reviewed changes

Comment thread libdd-trace-utils/src/span/vec_map.rs Outdated

Comment thread libdd-trace-utils/src/span/vec_map.rs

yannham added 2 commits May 28, 2026 15:09

fix: double log when deduping vec_map

5e118fe

test: fix failing test (vecmap order not guaranteed)

15e2e0e

yannham changed the title ~~feat(trace-utils): add dedup convenience to VecMap~~ feat(trace-utils)!: add dedup convenience to VecMap May 28, 2026

yannham mentioned this pull request May 28, 2026

refactor(span)!: use VecMap for meta, metrics and meta_struct for v04 spans #2043

Open

paullegranddc reviewed May 29, 2026

View reviewed changes

Comment thread libdd-trace-utils/src/span/vec_map.rs Outdated

refactor(trace-utils): store HashMap directly in DedupedVecMap::Owned

0996e9a

Avoid the redundant collect from HashMap back into Vec when deduping. The Owned variant now holds a HashMap<&K, &V> directly, removing one allocation and copy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

yannham requested a review from paullegranddc June 1, 2026 17:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(trace-utils)!: add dedup convenience to VecMap#2049

feat(trace-utils)!: add dedup convenience to VecMap#2049
yannham wants to merge 4 commits into
mainfrom
yannham/vecmap-as-dedup

yannham commented May 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented May 28, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented May 28, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented May 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

yannham commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check Results

📦 libdd-trace-utils - 602 warning(s)

Uh oh!

github-actions Bot commented May 28, 2026

Clippy Allow Annotation Report

Summary by Rule

Annotation Counts by File

Annotation Stats by Crate

About This Report

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Cargo Deny Results

📦 libdd-trace-utils - 4 error(s)

Uh oh!

datadog-datadog-prod-us1-2 Bot commented May 28, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dd-octo-sts Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yannham commented May 28, 2026 •

edited

Loading

github-actions Bot commented May 28, 2026 •

edited

Loading

📦 `libdd-trace-utils` - 602 warning(s)

github-actions Bot commented May 28, 2026 •

edited

Loading

📦 `libdd-trace-utils` - 4 error(s)

datadog-datadog-prod-us1-2 Bot commented May 28, 2026 •

edited by datadog-official Bot

Loading

codecov-commenter commented May 28, 2026 •

edited

Loading

dd-octo-sts Bot commented May 28, 2026 •

edited

Loading