chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642
chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642jbachorik wants to merge 20 commits into
Conversation
…ALLOC_PERTURB_ for glibc test runs Antagonist races tracer-style context propagation against virtual-thread carrier churn to target the ContextStorageMode.THREAD stale-carrier use-after-free. Since such heap corruption manifests silently until an unrelated later allocation, also turn on glibc's own corruption checks by default for chaos runs and ddprof-test suites (skipped on musl and whenever a sanitizer/allocator already owns malloc via LD_PRELOAD). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI Test ResultsRun: #29278209773 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-13 19:41:06 UTC |
J9 hits a pre-existing, unrelated glibc heap-corruption bug that MALLOC_CHECK_ now surfaces as a hard abort; gate it off until that's investigated separately.
Reliability & Chaos Results❌ 10 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124310859 ❌ chaos: profiler tracer gmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer gmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tracer gmalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer gmalloc amd64 25 0 3 temXchaos❌ chaos: profiler tracer jemalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer jemalloc amd64 25 0 3 temXchaos❌ chaos: profiler tracer tcmalloc aarch64 21 0 3 temXchaos❌ chaos: profiler tracer tcmalloc aarch64 25 0 3 temXchaos❌ chaos: profiler tracer tcmalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer tcmalloc amd64 25 0 3 temXchaos |
|
isRawPointer() only checked RAW_POINTER_MASK, letting raw J9 ASGCT BCIs that never went through encode() misroute into JVMSupport::resolve(), which asserts false for non-HotSpot VMs. Require ENCODED_MASK too.
It was registered in Main.java's antagonist factory but never added to either config's ANTAGONISTS string, so it silently never ran in CI.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1dd0fb1c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Include ACTIVE_PHASE_MILLIS in STALE_RACER_SLEEP_MILLIS so racers wake as the next active phase begins instead of ~3s early. Bound stopGracefully's total wait to the caller's timeout via a shared deadline instead of giving each join/tryAcquire the full budget independently.
…ntext via @trace driverLoop() called JavaProfiler.getInstance() directly, but JavaProfiler is a chaosCompileOnly dependency never bundled into chaos.jar and not visible to the app classloader (the agent's copy is shaded/relocated). The resulting NoClassDefFoundError is an Error, not an Exception, so it escaped the catch (Exception e) block and silently killed the driver thread — leaving activePhase stuck true and letting the sibling pinningChurnLoop run full-throttle for the rest of the test instead of its intended active/quiet duty cycle, which is the likely driver behind the OOM seen in reliability-chaos-aarch64 [profiler, gmalloc, 21.0.3-tem]. Rewrite the antagonist to only touch the tracer through @Trace-annotated methods (the same proven-safe dd-trace-api compileOnly pattern already used by TraceContextAntagonist), letting the real tracer and its virtual-thread instrumentation drive setContext/clearContext instead of calling com.datadoghq.profiler.* directly. Drops the custom-attribute/baggage simulation in favor of nested @trace hops plus forceUnmount and stale-buffer racer threads that still race OtelContextStorage's thread-scoped fallback across carrier churn. Verified with a 90s live run (JDK 21.0.11, profiler+tracer, gmalloc): no NoClassDefFoundError, no exceptions, completed cleanly with RC=0. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without a RUNTIME arg, chaos_check.sh silently proceeds with an empty value, which becomes --duration s in the chaos harness invocation and fails 15+ seconds later — after the JDK/agent/jar setup — with a NumberFormatException buried in the harness output, reported generically as "FAIL:Chaos harness crashed (RC=1)". Check for a missing RUNTIME right after arg parsing and fail immediately with a clear usage message. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit 3e379a7)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123961967 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
|
Not a review, but one thing that I can't quickly see from looking at the code: does it trigger actual migrations of vthreads across carriers? Because that is the fault scenario that led to the heap corruption. |
|
@rkennke It is trying to - but it is very probabilistic. It is trying to hold back carrier threads so new ones have to be spawned, then let the old one die so the vthread will have to be remounted but DBB still point to now invalid memory. |
J9
|
I see. If you have the time/nerve(/AI tokens) for it, you could perhaps try using a custom FJP where you can control the thread lifecycles. Probabilistic is fine, though, better than nothing. |
|
@rkennke That's a great idea. Thanks! |
Extract chaos_check.sh's logic into utils/run-chaos-harness.sh for local repro. Add a #ifdef DEBUG-only watchdog in ProfiledThread's free path (threadLocalData.cpp) that detects writes into freed OTel-context memory, and beef up VirtualThreadContextCascadeAntagonist's stale-carrier racing (continuous decoupled driver, randomized race window, dead-code removal). Fixes from review: Main.java antagonist-start try/finally, watcher thread signal-mask inheritance, racer-loop backoff under saturation, redundant frame_ut.cpp assertions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
watchFreedContextMemory() deliberately read freed ProfiledThread memory to detect stale writes, tripping ASan/TSan on every free (e.g. in forced_unwind_ut) unrelated to the actual race under investigation.
Benchmark Results (commit b71ca62)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124279353 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10191 ms (21 iters) | ✅ 10399 ms (21 iters) | ≈ +2% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8940 ms (24 iters) | ✅ 8819 ms (24 iters) | ≈ -1.4% (±9.6%) | — / — |
| finagle-chirper | 21 | ✅ 5958 ms (33 iters) | ✅ 5999 ms (33 iters) | ≈ +0.7% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5524 ms (36 iters) | ✅ 5491 ms (36 iters) | ≈ -0.6% (±25%) | |
| fj-kmeans | 21 | ✅ 2746 ms (68 iters) | ✅ 2742 ms (68 iters) | ≈ -0.1% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2737 ms (69 iters) | ✅ 2836 ms (66 iters) | 🔴 +3.6% | — / — |
| future-genetic | 21 | ✅ 2083 ms (89 iters) | ✅ 2065 ms (90 iters) | ≈ -0.9% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2041 ms (90 iters) | ✅ 2141 ms (87 iters) | 🔴 +4.9% | — / — |
| naive-bayes | 21 | ✅ 1291 ms (133 iters) | ✅ 1255 ms (136 iters) | ≈ -2.8% (±31.7%) | — / — |
| naive-bayes | 25 | ✅ 983 ms (174 iters) | ✅ 999 ms (171 iters) | ≈ +1.6% (±31.9%) | — / — |
| reactors | 21 | ✅ 16505 ms (15 iters) | ✅ 16271 ms (15 iters) | ≈ -1.4% (±8.3%) | — / — |
| reactors | 25 | ✅ 18606 ms (14 iters) | ✅ 18677 ms (15 iters) | ≈ +0.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2094 / 2059 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 5 | 2307 / 2237 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8584 / 8425 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 8769 / 8940 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1252 / 1221 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1320 / 1251 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2917 / 2984 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2900 / 2964 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 3507 / 3542 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3509 / 3455 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1682 / 1599 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1520 / 1920 | ✅ / ✅ | ✅ / ✅ |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b71ca62c0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The skip's only justification was PROF-15360, closed as invalid in favor of PROF-15364 (isRawPointer() false-positive, fixed by this PR). No separate J9 heap-corruption issue is on record, so J9 now gets the same MALLOC_CHECK_/MALLOC_PERTURB_ coverage as glibc/HotSpot.
Confirmed case 2. Traced the skip's history: the commit that added it ( |
Benchmark Results (commit 1d054cc)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124288733 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10249 ms (21 iters) | ✅ 10378 ms (21 iters) | ≈ +1.3% (±10.7%) | — / — |
| akka-uct | 25 | ✅ 8766 ms (24 iters) | ✅ 8849 ms (24 iters) | ≈ +0.9% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5973 ms (33 iters) | ✅ 5999 ms (33 iters) | ≈ +0.4% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5498 ms (36 iters) | ✅ 5448 ms (36 iters) | ≈ -0.9% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2740 ms (68 iters) | ✅ 2777 ms (67 iters) | ≈ +1.4% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2813 ms (66 iters) | ✅ 2802 ms (66 iters) | ≈ -0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2042 ms (90 iters) | ✅ 2218 ms (84 iters) | 🔴 +8.6% | — / — |
| future-genetic | 25 | ✅ 2098 ms (88 iters) | ✅ 2010 ms (93 iters) | 🟢 -4.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1279 ms (134 iters) | ≈ +0.8% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1007 ms (170 iters) | ≈ -0.5% (±31.3%) | — / — |
| reactors | 25 | ✅ 18623 ms (15 iters) | ✅ 18109 ms (15 iters) | ≈ -2.8% (±5.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2035 / 2016 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2000 / 2280 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8876 / 8515 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8312 / 8747 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1277 / 1255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1287 / 1258 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2993 / 3089 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2898 / 2898 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3554 / 3472 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3475 / 3493 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1903 / 1926 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit fb6a1e8)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124310879 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10197 ms (21 iters) | ✅ 10305 ms (21 iters) | ≈ +1.1% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8925 ms (24 iters) | ✅ 8774 ms (24 iters) | ≈ -1.7% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5925 ms (33 iters) | ✅ 5973 ms (33 iters) | ≈ +0.8% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5436 ms (36 iters) | ✅ 5402 ms (36 iters) | ≈ -0.6% (±23.7%) | |
| fj-kmeans | 21 | ✅ 2651 ms (71 iters) | ✅ 2759 ms (68 iters) | 🔴 +4.1% | — / — |
| fj-kmeans | 25 | ✅ 2837 ms (66 iters) | ✅ 2843 ms (66 iters) | ≈ +0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2038 ms (91 iters) | ✅ 2134 ms (87 iters) | 🔴 +4.7% | — / — |
| future-genetic | 25 | ✅ 2085 ms (89 iters) | ✅ 2107 ms (88 iters) | ≈ +1.1% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1234 ms (138 iters) | ✅ 1249 ms (136 iters) | ≈ +1.2% (±32.6%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1023 ms (167 iters) | ≈ +0.7% (±31.6%) | — / — |
| reactors | 21 | ✅ 16230 ms (15 iters) | ✅ 16350 ms (15 iters) | ≈ +0.7% (±8.4%) | — / — |
| reactors | 25 | ✅ 18749 ms (15 iters) | ✅ 18774 ms (15 iters) | ≈ +0.1% (±5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2022 / 1981 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2389 / 2312 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 8854 / 8395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8242 / 8401 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 8 | 1286 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 1293 / 1264 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2929 / 2992 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 2895 / 2906 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3507 / 3474 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 3485 / 3437 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1966 / 1937 | ✅ / ✅ | ✅ / ✅ |
…IER mode Without --add-exports java.base/jdk.internal.misc=ALL-UNNAMED, OtelContextStorage silently falls back to ContextStorageMode.THREAD, which vthread-context-cascade was built to catch: virtual-thread context buffers pinned to a since-freed carrier's ProfiledThread, corrupting the heap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Root-caused the reliability crashes reported in Confirmed the crash requires the ddprof agent (the same antagonist run with no The harness was never passing Worth calling out: |
|
@copilot resolve the merge conflicts in this pull request |
Benchmark Results (commit c7bf65c)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124374293 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10216 ms (21 iters) | ✅ 10186 ms (21 iters) | ≈ -0.3% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8851 ms (24 iters) | ✅ 8864 ms (24 iters) | ≈ +0.1% (±9.9%) | — / — |
| finagle-chirper | 21 | ✅ 6035 ms (33 iters) | ✅ 5965 ms (33 iters) | ≈ -1.2% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5451 ms (36 iters) | ✅ 5420 ms (36 iters) | ≈ -0.6% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2791 ms (67 iters) | ✅ 2784 ms (67 iters) | ≈ -0.3% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2835 ms (66 iters) | ✅ 2822 ms (66 iters) | ≈ -0.5% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2112 ms (88 iters) | ✅ 2119 ms (88 iters) | ≈ +0.3% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2091 ms (89 iters) | ✅ 1947 ms (95 iters) | 🟢 -6.9% | — / — |
| naive-bayes | 21 | ✅ 1325 ms (129 iters) | ✅ 1239 ms (137 iters) | ≈ -6.5% (±31.7%) | — / — |
| naive-bayes | 25 | ✅ 1023 ms (167 iters) | ✅ 1005 ms (170 iters) | ≈ -1.8% (±31.4%) | — / — |
| reactors | 21 | ✅ 16789 ms (15 iters) | ✅ 15597 ms (16 iters) | 🟢 -7.1% | — / — |
| reactors | 25 | ✅ 18388 ms (15 iters) | ✅ 19200 ms (14 iters) | ≈ +4.4% (±4.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2004 / 1877 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 2212 / 2130 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 9142 / 8336 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8043 / 8491 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1278 / 1245 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1289 / 1272 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2963 / 2969 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2990 / 2839 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3537 / 3505 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 3 | 3488 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1779 / 1596 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1851 / 2013 | ✅ / ✅ | ✅ / ✅ |
Resolved by merging |
Benchmark Results (commit 2e34507)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124382794 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
profiler+tracer chaos runs OOMKilled on aarch64 (exit 137) with gmalloc/jemalloc but not tcmalloc, which already had aggressive-decommit tuning for the same thread-churn RSS-inflation issue. Apply the equivalent tuning to the other two allocators.
What does this PR do?:
Adds a new
vthread-context-cascadechaos antagonist that races tracer-stylecontext propagate/activate/restore semantics against virtual-thread carrier
churn, targeting the
ContextStorageMode.THREADstale-carrierDirectByteBufferuse-after-free. Also defaultsMALLOC_CHECK_/MALLOC_PERTURB_for glibc-backed test runs (chaos reliability runner andthe shared
ddprof-testGradle test-task configuration), turning silentheap corruption into an immediate, attributable
SIGABRTinstead of acrash much later in an unrelated allocation.
Also fixes a real J9-only bug (PROF-15364) uncovered while chasing the
MALLOC_CHECK_default's fallout:FrameType::isRawPointer()(frame.h)misidentified raw, unencoded J9 ASGCT BCIs as HotSpot raw-pointer-encoded
frames, routing them into
JVMSupport::resolve()'sassert(false)fornon-HotSpot VMs — a debug-only SIGABRT. This is required for this PR's own
J9 debug CI config to pass; outside this branch/config (i.e. before the
MALLOC_CHECK_default made the J9 debug suite deterministic) it wasn'teasily reproducible.
Motivation:
While building the
vthread-context-cascadereproducer it became clearthat this class of use-after-free bug corrupts glibc heap/tcache metadata
silently and only manifests as a crash far later, in an unrelated
allocation call — making it unlikely to be caught within a short chaos
run or CI test window. Enabling glibc's own heap-corruption checks by
default closes that gap cheaply, without needing a sanitizer build (which
can't run against a live JVM, see
reference_sanitizers_no_jvm).Additional Notes:
The
MALLOC_CHECK_/MALLOC_PERTURB_defaults are skipped on musl (noMALLOC_CHECK_support) and whenever a sanitizer/allocator alreadyreplaces malloc via
LD_PRELOAD(ASan/TSan/tcmalloc/jemalloc configs),mirroring the existing
gmallocvstcmalloc/jemallocmutualexclusivity in
chaos_check.sh.The
isRawPointer()fix (frame.h) makes the check also requireENCODED_MASK(bit 20) before trustingRAW_POINTER_MASK(bit 30) —RAW_POINTER_MASKis only ever set byencode(..., rawPointer=true),which is HotSpot-only and always sets
ENCODED_MASKtoo, so a rawunencoded ASGCT BCI (used on J9) can no longer false-positive just because
it happens to have bit 30 set. PROF-15360 (originally filed as suspected
"silent glibc heap corruption on J9") is closed in favor of PROF-15364,
which documents this corrected root cause.
How to test the change?:
./gradlew :ddprof-stresstest:chaosJarbuilds cleanly with the new antagonist registered../gradlew :build-logic:conventions:compileKotlincompiles cleanly with theProfilerTestPluginchange.-Dddprof.debug.context.storage.mode=thread, tuned virtual-thread scheduler flags,MALLOC_CHECK_=3/MALLOC_PERTURB_set) for several minutes; no crash reproduced yet in this window — the antagonist is being landed as a reliability-suite addition to keep exercising this race over longer soak runs, not as a guaranteed-repro smoke test.FrameTypeIsRawPointerTest.FalseForRawAsgctBciWithBit30SetButNoEncodedMarkertoframe_ut.cpp(regression test for theisRawPointer()fix);ddprof-lib:gtestDebug_frame_utpasses 18/18, including the pre-existingTrueWhenBit30IsSetcase (no regression on the legitimate HotSpot path)../gradlew :ddprof-lib:buildDebug -x testsucceeds.glibc/jdk8/j9/aarch64debug container (Assertion 'false && "Should not reach here"' failedatjvmSupport.inline.h:34) before this fix — this is the repro that led to filing and fixing PROF-15364.For Datadog employees:
If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.This PR doesn't touch any of that.
JIRA: PROF-15364 (fixed by this PR); closes [PROF-15360]
Unsure? Have a question? Request a review!