Fix pod proxy shutdown retirement race

[luke-lombardi]

Summary

• retire pod backends from proxy discovery before scheduler shutdown
• propagate STOPPING state across gateway replicas and prune idle transports
• queue concurrent cold-start requests when another gateway holds the scaling lock
• preserve explicit force-stop semantics while rechecking connections for automatic scale-down

Root cause

The autoscaler could select an idle pod for shutdown while its route and HTTP keepalive transport were still published. A request arriving at the shutdown boundary could reuse that transport and receive an immediate 502 instead of waiting for a checkpoint restore. Discovery could also republish the still-RUNNING container before Scheduler.Stop committed STOPPING.

Validation

• go test ./pkg/abstractions/common ./pkg/abstractions/pod -count=1
• go test -race ./pkg/abstractions/pod -count=1
• staging: repeated untouched Qwen checkpoint restores with exact chat completions, including shutdown-boundary and concurrent request waves; no 500/502 responses
• restored staging to two stock gateway 0.1.722 replicas after Okteto testing

---

Summary by cubic

Fixes a shutdown race in the pod proxy that caused 502s at scale-down. The proxy now retires backends before stopping containers, propagates STOPPING across replicas, and queues cold-starts when another gateway holds the scaling lock.

• Bug Fixes
  • Retire containers from discovery and remove idle transports before Scheduler.Stop to prevent keepalive reuse.
  • Track retiring container IDs in PodProxyBuffer and filter them during discovery; allow canceling retirement if needed.
  • Propagate STOPPING via container events; prune transports on state change; check both local and shared connection counts.
  • Reject automatic stop when connections exist; keep explicit force-stop behavior.
  • On scaling lock contention, trigger autoscaler and allow requests to queue instead of erroring.

^{Written for commit 8cbac86. Summary will update on new commits.}