Skip to content

fix(sglang): never skip kill_process_tree when router removal fails in shutdown#32

Open
JunzheJoe wants to merge 1 commit into
zhenyu/m11-mvp-testfrom
joe/shutdown-router-leak
Open

fix(sglang): never skip kill_process_tree when router removal fails in shutdown#32
JunzheJoe wants to merge 1 commit into
zhenyu/m11-mvp-testfrom
joe/shutdown-router-leak

Conversation

@JunzheJoe

Copy link
Copy Markdown

What

Review finding F8-SHUTDOWN-ROUTER-LEAK (F1-F12 code review, leak-level — reported per the review scope's miles exception: crash / data corruption / resource leak).

SGLangEngine.shutdown()'s miles-router / old-router branch posts /remove_worker with no exception guard. When the router is already dead — which is exactly the failure-cleanup scenario where RolloutManager.shutdown_hard drives this method (router crash → training fails → driver finally → shutdown_hard) — requests raises ConnectionError, shutdown() aborts, and kill_process_tree at the bottom never runs.

The caller can't compensate: rollout.py's shutdown_hard catches the exception and falls back to ray.kill(handle), but killing the Ray actor does not terminate its spawned SGLang server child processes (rollout.py's own comment: "CUDA context lives in the SGLang server child processes"). The orphaned tree leaves Ray's process hierarchy, so even ray stop may not reclaim it — the servers hold their VRAM until a manual pkill, and subsequent runs OOM.

Fix

Bring the miles-router / <0.3.0 branches to parity with the >=0.3.0 branch (which already had a try/except): wrap the whole router-removal block, warn on failure, and always run kill_process_tree. Router removal becomes best-effort — shutdown's job is killing the server tree, and a dead router must not prevent that. Inner version-branch logic is unchanged (indentation-only diff inside the try).

Testing

  • ast.parse clean; no unit test — exercising this path requires stubbing requests + a live process tree, and the failure mode is a straight control-flow skip that the diff makes structurally impossible.
  • Behavior change is strictly monotone: previous behavior on router failure was raise + leak; new behavior is warn + kill. The standalone path benefits identically.

…n shutdown

The miles-router / old-router branch of SGLangEngine.shutdown posted
/remove_worker without any exception guard. When the router is already
dead — which is exactly the failure-cleanup scenario where
RolloutManager.shutdown_hard drives this method — requests raises
ConnectionError, the method aborts, and kill_process_tree never runs.

ray.kill on the engine actor does not terminate its child processes,
and the CUDA context lives in the SGLang server tree, so the orphaned
servers keep their VRAM until a manual pkill; subsequent runs OOM.
The >=0.3.0 router branch already had a try/except guard — this brings
the other branches to parity and makes router removal best-effort:
warn on failure, always kill the process tree.
@JunzheJoe JunzheJoe requested a review from taoluo July 7, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant