CI: graph-lifetime tests flaky on free-threaded 3.14t jobs since #2371

[leofang]

Describe the bug

Since #2371 merged (2026-07-15 16:55 UTC), CI is flaky on free-threaded Python 3.14t test jobs: four tests in cuda_core/tests/graph/test_graph_definition_lifetime.py intermittently fail with

AssertionError: condition not satisfied within 5.0s

raised by the _wait_until polling helper (_FINALIZE_TIMEOUT = 5.0):

• test_kernel_args_buffer_lifetime (added in cuda.core: keep kernel-argument objects alive in graph kernel nodes #2041)
• test_memcpy_buffer_allocations_released_after_graph_destroyed (cuda.core: graph slot table for node attachment lifetimes #2280)
• test_slot_owners_released_after_graph_destroyed (cuda.core: graph slot table for node attachment lifetimes #2280)
• test_pending_call_queue_saturation_preserves_cleanup (cuda.core: defer graph user-object payload cleanup #2371)

Impact (2026-07-15 → 2026-07-21)

Survey of all Actions runs 2026-06-15 → 2026-07-22 (246 failed/re-run runs classified by job-log signature):

• 16 of 77 CI runs (~21%) hit this signature, including 5 runs on main (push + scheduled). 11 were on PRs touching nothing near graph code (e.g. Pathfinder: add support on cutensorMp library loading #2374, Raise OverflowError on out-of-range c_int/c_byte kernel arguments #2402).
• 7 of the 16 were re-run to green — one needed 6 attempts — so red-run counts understate it.
• Zero occurrences in ~300 completed runs 2026-06-15 → 2026-07-14. cuda.core: graph slot table for node attachment lifetimes #2280 (merged 2026-07-07) ran clean for 8 days; the first hit is 4.5 h after cuda.core: defer graph user-object payload cleanup #2371 merged (2026-07-15 21:26 UTC, run 29451966527).
• Only 3.14t jobs are affected — across win-64 / linux-64 / linux-aarch64, CUDA 12.9.1 / 13.0.2 / 13.3.0, local & wheels, T4 / L4 / A100. Never observed on regular-GIL jobs.

Failing jobs (all 18 instances)

Failing CI pipeline	CI log
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-15 · PR 2374
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-16 · PR 2242
Test linux-aarch64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100	2026-07-17 · PR 2379
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-17 · main
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-17 · main
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-19 · main
Test linux-64 / Python 3.14t, CUDA 12.9.1 (local), GPU t4	2026-07-20 · PR 2390
Test linux-64 / Python 3.14t, CUDA 13.0.2 (local), GPU l4	2026-07-20 · PR 2242
Test linux-aarch64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100	2026-07-20 · main
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-20 · PR 2242
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-20 · PR 2394
Test linux-64 / Python 3.14t, CUDA 12.9.1 (local), GPU t4	2026-07-21 · PR 2398
Test linux-64 / Python 3.14t, CUDA 12.9.1 (local), GPU t4	2026-07-21 · PR 2402
Test linux-aarch64 / Python 3.14t, CUDA 13.3.0 (local), GPU l4	2026-07-21 · main
Test linux-aarch64 / Python 3.14t, CUDA 13.3.0 (local), GPU l4	2026-07-21 · PR 2383
Test win-64 / Python 3.14t, CUDA 12.9.1 (local), GPU l4 (TCC)	2026-07-21 · PR 2390
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-21 · PR 2392
Test win-64 / Python 3.14t, CUDA 13.0.2 (wheels), GPU a100 (MCDM)	2026-07-21 · PR 2398

(Excluded from the count: graph PRs failing their own branches — #2280, #2333, #2357 — and the 2026-07-17 gb300 nightly, which failed machine-wide with ValueError: Cannot set NULL context as current and looks unrelated.)

How to Reproduce

No deterministic local repro; it fires in roughly 1 in 5 CI runs on any 3.14t GPU test job since 2026-07-15. The four tests above are the ones to stress.

Expected behavior

Graph-lifetime cleanup assertions pass reliably on free-threaded builds.

Likely mechanism

#2371 defers user-object payload cleanup through a pending-call queue. On free-threaded Python (deferred reference counting), on a loaded runner, cleanup can lag past the 5 s _FINALIZE_TIMEOUT budget, so assertions observing release (weakref death, allocation counters, queue drain) time out. Three of the four tests predate #2371 and only began flaking once it landed.

Possible directions:

• Raise _FINALIZE_TIMEOUT on free-threaded builds (it is already 30 s under compute-sanitizer).
• Make _wait_until actively drive the pending-call queue instead of only gc.collect() + sleep.
• Bound cleanup latency in the implementation (e.g. drain the queue opportunistically on API calls).

-- Leo's bot