[live-migration] reconnect init stdout/stderr relay from guest on host disconnect

[Harsh Rawat (rawahars)]

Summary

When the host end of a container init process's stdout/stderr vsock connection drops mid-stream, the guest previously blocked forever on write — the relay goroutine parked and the container stalled once its pipe filled.

This change makes the guest transparently re-dial the same vsock port every 100 ms, replay recently-written bytes, and resume the relay, so the stream survives a host disappearance with no log loss.

Implementation (scoped to the init process's stdout/stderr only):

• A reconnecting-connection decorator wraps the connections in Container.Start. On a disconnect-class write error (EPIPE/ECONNRESET/ENOTCONN) it re-dials the same port on a fixed 100 ms cadence until it reconnects.
• To avoid losing bytes that write() accepted but the host never consumed (they can sit undelivered in the host/guest socket buffers when the drop happens), the decorator keeps a bounded replay buffer (128 KiB) of the most-recently-written bytes and re-sends it on every reconnect. This recovers the in-flight bytes at the cost of re-sending some already-delivered bytes (duplicates), which is acceptable for logs.
• Retries are bounded by container lifetime (Kill/Delete), so a stuck reconnect can never block relay teardown or process-exit reporting. stdin, exec processes, and external processes are unchanged as they are not applicable for LM scenarios.

Why 128 KiB of replay buffer

The replay buffer must be at least the worst-case in-flight window: everything that can be buffered between the guest's write() returning and the host relay reading it. When the connection drops abruptly — notably at the live-migration blackout, where the host force-aborts its hvsocket endpoint and discards its receive buffer — that whole window is lost, so replay must cover it. It is the sum of two fixed OS buffers:

• Guest outbound hv_sock send ring: 24 KiB — Linux hv_sock RINGBUFFER_HVS_SND_SIZE (6×4 KiB). The vsock dialer never raises SO_SNDBUF, so it stays at this floor.
• Host hvsocket receive buffer: 64 KiB — Windows AFD default SO_RCVBUF (65536).

Worst case ≈ 88 KiB by construction. Therefore, 128 KiB rounds up with headroom. Both terms are hard OS caps that do not grow with container runtime or output rate, so a bigger buffer buys nothing, and a smaller one risks loss if the host log sink stalls and both buffers are full at the instant of a drop.

Log Loss

Measured empirically (host stdout connection dropped mid-stream via a test hack, then restored). With the replay buffer the stream is lossless at every write rate and every outage duration — every line is present after reconnect, with no torn lines. Because the loss window is a fixed, OS-bounded quantity that 128 KiB always exceeds, there is no rate or outage length at which loss occurs.

The only cost is a bounded burst of duplicate lines. On reconnect the guest re-sends its retained tail (up to 128 KiB); it re-sends the whole tail because the guest has no feedback on how much the host actually consumed before the drop (no application-level ACK). Only the true in-flight window (≤ ~97 KiB, usually far less) was genuinely lost; the remainder is duplicate the host already had.

Outage duration doesn't affect loss: output produced while the host is away is held in the guest pipe — the container simply blocks once the pipe fills — and streams normally once the relay resumes; the replay buffer separately recovers the bytes that were in the socket buffers at the instant of the drop.

Duplicate volume vs. write rate

Loss is zero at every rate. This table shows how quickly the replay tail reaches its 128 KiB maximum — the ceiling on duplicates re-sent per reconnect. Before that point, the tail (and thus the duplicates) is just everything written so far.

128 KiB ≈ ~16–18 K lines (at ~7–8 bytes/line)

Cadence	Rate	Time to fill 128 KiB
Full throttle	~30 MB/s	< 1 s
1 ms	~300 lines/s	~1 min
10 ms	~100 lines/s	~3 min
100 ms	~10 lines/s	~28 min
1 s	~1 line/s	~4.5 hours

Once a container has emitted more than 128 KiB total (essentially any real workload within minutes), each reconnect re-sends the full 128 KiB, almost all of which is duplicate. Trimming that further would require a host→guest ACK of the last durably-logged byte, which is out of scope here.

Additional Validation

• Ran a scenario by hacking the shim on top of this change. We live-migrated a pod from source to destination, where the container exited while the gcs reconnection was still pending after a successful VM resume. When the GCS connection was re-established, all logs prior to container exit were flushed and an exit notification was generated on the resumed pod.

[Shreyansh Sancheti (shreyanshjain7174)]

If replay on the newly accepted connection fails, redial closes it and tries another dial. The host ioChannel listener accepts once and then closes, so the second dial cannot succeed. I reproduced this on the combined branch with a one-shot transport: replay failure caused another dial against the consumed listener. Can we return the replay failure instead of retrying indefinitely, and add this case to the test?

[Harsh Rawat (rawahars)]

Unfortunately, we cannot stop the guest from retrying even if the host is a one-shot listener.

Consider a scenario wherein we perform sequential migrations of a pod from VM1 -> VM2 -> VM3. In both the migrations, a new host listener will be attached but guest does not know if the drop was due to listener getting dropped or a migration.

Today, if the host listener for a container stdio errors out due to any reason, the error is logged on the host and guest side connection goes dark. The stdio just stops streaming. With this change, the delta would be retries from within the guest to dial to a non-existent connection.

Even in the path where listener errors out outside LM, the redial loop should have negligible CPU perf impact since these syscalls are cheap.

[Shreyansh Sancheti (shreyanshjain7174)]

Agreed. I overlooked that a later migration/resume creates a fresh listener, so the guest must continue retrying until stopped LGTM!