[NVBUG-6448152][test] isolate PP rendezvous from global commit

[chienchunhung]

Diagnostic only — do not merge. This historical-tree experiment isolates the exposed PP rendezvous cost in NVBUG#6448152 while preserving the global terminal decision and existing resource lifetime.

Question

Does replacing #15139's synchronous CTX TP/PP outcome collectives with asynchronous point-to-point agreement recover the adjacent pre-#15139 throughput, even when request/KV resources remain retained until global commit?

Design

• PP rank P-1 is the coordinator; [NVBUG-6448152][test] trace disagg transfer lifecycle #16567 showed it naturally reaches the terminal-status point last.
• Each rank publishes one immutable {request, success|failure} vote without waiting.
• The coordinator reduces the votes and broadcasts one idempotent global commit.
• Failure has precedence once all immutable votes have arrived; this diagnostic reducer waits for the full vote set for both outcomes. A production protocol could safely short-circuit the coordinator's failure decision, provided followers buffer that commit until their own local transfer is quiescent.
• Normal status polling progresses the protocol but never waits for the other PP ranks. Only the explicit block-all/shutdown path drains outstanding commits.
• Final request state, response publication, and KV/resource reclamation remain gated by global commit, matching [TRTLLM-12721][feat] Add disagg transfer state consensus #15139 semantics.
• One startup PP allgather negotiates protocol version/effective mode. There is no collective on the per-request hot path.

This preserves a common ordered decision across ranks, but application is asynchronous rather than simultaneous in wall-clock time.

Exact experiment

• Historical parent: c58a3ac9be5ca30d69a5d326f250768b021f97f2 ([TRTLLM-12721][feat] Add disagg transfer state consensus #15139 treatment plus the common [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737 sender fix)
• Experiment commit: dc06e82a182671a0d9626243d660a44a3b73bd1d
• Experiment tree: 68e92a26956da25a659860082da887567b9d36c3
• CI head: 5e2498b46a71f53f39fd641b3d5566cd4dd11f4d
• No-tree-change CI merge parent: 176041ba15f25fda876ccfa209d04896a003dc68
• [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737 dataTransceiver.cpp blob preserved: b3eb1c0f984394e798993708bebcabe885ea1644
• Activation: TRTLLM_NVBUG_6448152_CTX_COORDINATOR_CONSENSUS=1
• Qualification: C++ transceiver, NIXL, MPI control plane, TP1/CP1/PP>1, attention DP off. The shared GEN PP1 service remains on its existing path.

Targeted validation

Stage: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

Selector: disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL

A throughput result is valid only with 512/512 successful requests. Compare with:

• matched pre-[TRTLLM-12721][feat] Add disagg transfer state consensus #15139 + [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737 control [NVBUG-6448152][test] control #15139 with sender wakeup fix #16565: 1557.83 output tok/s
• matched [TRTLLM-12721][feat] Add disagg transfer state consensus #15139 + [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737 treatment [NVBUG-6448152][test] test #15139 with sender wakeup fix #16566: timeout; raw 598.42 is censored
• trace [NVBUG-6448152][test] trace disagg transfer lifecycle #16567: pre-teardown approximately 818 tok/s, with PP entry skew localized
• unsafe local-only CTX control [NVBUG-6448152][test] isolate CTX consensus regression #16572: 1564.71 tok/s

Risks / limitations

• Diagnostic protocol, not a production-ready replacement: shutdown uses a bounded abort epoch for asymmetric process failure.
• Global commit may reach ranks at different times; the result is identical and ordered, not applied simultaneously.
• This arm deliberately retains KV until commit, so a low result does not by itself disprove rendezvous overhead; compare with [NVBUG-6448152][test] measure local-quiescence reclamation #16581, which keeps identical vote traffic and changes only safe local-success KV reclamation.

Dependency and experiment graph

flowchart LR
    C["#16565<br/>pre-#15139 + #15737<br/>1557.83 tok/s"]
    T["#16566 / #16567<br/>post-#15139 + #15737<br/>~818 tok/s before teardown"]
    A["#16580<br/>async PP agreement<br/>global KV retention"]
    B["#16581<br/>same agreement traffic<br/>local-success KV reclamation"]
    L["#16572<br/>local-only unsafe control<br/>1564.71 tok/s"]
    C -->|"#15139 only"| T
    T --> A
    A --> B
    T --> L

Loading

Interpretation: A at least 90% of #16565 means exposed rendezvous dominated. A low but B at least 90% means retained KV/admission was the remaining dominant cost. Partial recovery in both means both contribute. Any failed request censors throughput.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60200 [ run ] triggered by Bot. Commit: 5e2498b Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A30-CPP-Post-Merge-1,A30-CPP-Post-Merge-2,DGX_H100-4_GPUs-CPP-1,DGX_H100-4_GPUs-PyTorch-Others-1,DGX_H100-4_GPUs-PyTorch-Others-2"