fix(envoy-client): ack terminating stop commands so pegboard-envoy stops replaying them

[abcxff]

No description provided.

[abcxff]

Stack for rivet-dev/actors

• #5565 fix(envoy-client): ack terminating stop commands so pegboard … 👈
• main

Get stack: forklift get 5565
Push local edits: forklift submit
Merge when ready: forklift merge 5565

_{change zrklppqy}

[railway-app]

🚅 Deployed to the actors-pr-5565 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
website	❌ Build Failed (View Logs)	Web	Aug 13, 2026 at 9:28 pm
kitchen-sink	😴 Sleeping (View Logs)	Web	Aug 13, 2026 at 5:56 pm
mcp-hub	✅ Success (View Logs)	Web	Aug 12, 2026 at 7:34 pm
frontend-inspector	❌ Build Failed (View Logs)	Web	Aug 12, 2026 at 7:34 pm
ladle	❌ Build Failed (View Logs)	Web	Aug 12, 2026 at 7:34 pm
frontend-cloud	❌ Build Failed (View Logs)	Web	Aug 12, 2026 at 7:34 pm

[claude]

Review

Clean, well-motivated fix. Sourcing acks from processed_command_idx (which survives remove_actor) instead of the live ctx.actors map correctly closes the bug where a fast-stopping actor's checkpoint never made it into an ack before the entry was removed, causing pegboard-envoy to replay the stop forever. The three new tests cover the core scenarios well (immediate ack, retained-on-send-failure + replay retry, unknown-actor stop).

A few things worth a look:

1. The immediate ack sweeps the entire processed_command_idx map, not just the stop's own checkpoint.
batch_has_stop only gates whether to fire send_command_ack_inner(ctx, false), but that function still builds last_command_checkpoints from every entry in ctx.processed_command_idx (commands.rs:108). So any batch containing a stop, even a duplicate/replayed one, now also immediately acks and clears (server-side) the replay safety net for unrelated, just-processed CommandStartActors in the same batch, rather than waiting for the 5-minute periodic tick (ACK_COMMANDS_INTERVAL_MS). The existing TODO above (commands.rs:139-148) already documents a narrow race where an ack can be committed by pegboard-envoy before the local dedup map reflects it; this change increases how often that ack fires for freshly-started actors, shrinking the crash-recovery window (envoy process dies right after a start is acked but before the actor is durably initialized) compared to before. Worth confirming this is intentional/acceptable, since it's a meaningful behavior change beyond "ack stops promptly", or scope the immediate ack to just the stop checkpoints if unintended.

2. ActorEntry::last_command_idx is now dead.
After switching ack sourcing to processed_command_idx, nothing in the crate reads entry.last_command_idx anymore (still written in commands.rs:74 and via insert_actor's last_command_idx param, but never read, confirmed via grep across the crate). Since the field/param are pub, this won't trigger a dead_code warning, but it's vestigial after this refactor. Consider removing it (and the insert_actor parameter) unless it's meant to stay for a future purpose.

3. Minor comment accuracy nit.
The comment at commands.rs:90-92 ("Ack a stop immediately since its actor is removed before the periodic tick") implies removal happens synchronously with the stop command. Looking at events.rs:14-36, remove_actor is actually only called later, when the actor's own Stopped state-update event arrives with received_stop already set, a separate, later event. Not wrong in effect (removal can still race ahead of the 5-minute tick), but slightly imprecise about when/why; might be clearer as "...may be removed before the next periodic tick."

4. Test coverage suggestion.
Given point 1, a test pinning the "a stop in the batch also flushes unrelated pending checkpoints (e.g., a start for a different actor in the same or a prior batch)" behavior would help make that scope-widening explicit and regression-proof, since it's the main behavioral change here beyond the targeted stop-ack fix.

Nothing blocking, the core fix is sound and matches the stated goal (stop pegboard-envoy from replaying acked-but-forgotten stops). Points 1 and 2 are the ones I'd want a second opinion on before merging.

[claude]

Review

Traced the full fix against the pre-PR version of commands.rs and the EnvoyContext/events.rs call sites to confirm the root cause and the fix's correctness.

Root cause confirmed: remove_actor (called from handle_send_events in events.rs once a stopped actor's ActorStateStopped event is processed) can fire well before the next 5-minute send_command_ack tick. Since the old send_command_ack only sourced checkpoints from ctx.actors, a terminal stop command for an already-removed actor was never acked, so pegboard-envoy kept replaying it forever. The fix addresses this correctly:

• Immediate, scoped ack of just the stopped actors right after handle_commands processes them (send_stop_command_acks), sourced from processed_command_idx, which persists across remove_actor.
• send_command_ack's periodic tick now also merges processed_command_idx (not just live ctx.actors) as a fallback, so a failed immediate send is recovered on the next tick.

Correctness walk-through:

• stopped_actors is collected from the raw batch before the dedup loop, so replayed-but-skipped stops are still re-acked. I verified the dedup invariant holds: whenever a stop is skipped as a duplicate, processed_command_idx is guaranteed to already contain that key (that's the precondition for the skip branch), so send_stop_command_acks's lookup never misses an entry.
• send_stop_command_acks intentionally does not clear dedup state, deferring to the periodic tick, consistent with the existing race-window TODO about acking before pegboard-envoy commits the clear_range. The updated comment correctly extends that TODO to cover the new removed-actor case.
• The entry.last_command_idx vs processed_command_idx merge via .max(...) in send_command_ack is a no-op for live actors (both are always kept in lockstep) and only matters for the removed-actor case it's meant to cover.

Test coverage: the four new tests cover the essential scenarios well: immediate ack, failed-send retry recovered via replay, stop-for-unknown-actor (never tracked/already gone), and periodic re-ack of a still-live actor. One gap: no test exercises a batch with multiple stopped actors at once (verifying the HashMap-based batching in send_stop_command_acks produces one ack per actor). Not a correctness concern given the logic, but would be a cheap addition.

Style/conventions: tabs, comment style (full sentences, no em dashes), import placement, and error handling all match CLAUDE.md conventions. No unwrap/panic! introduced, no new dependencies, no protocol/wire-schema changes (so no vbare version bump needed, this is client-side ack bookkeeping only). This targets rivet-envoy-client/pegboard-envoy, the current (non-deprecated) actor-hosting path, so no parity work against the deprecated runner is required.

Overall: a well-reasoned, correctly-scoped fix with good self-documentation of the remaining known race (extends the existing TODO). No blocking issues found.