edgeHub: resume store-and-forward promptly after reconnect

[jlian]

Summary

• Bridge stable Device SDK connection-status changes into edgeHub's connectivity manager after a 2-second debounce.
• Surface recovery from both Trying -> Connected and Disconnected -> Connected without changing the existing DeviceConnected event semantics.
• Interrupt a parked endpoint FSM retry timer when connectivity recovers so store-and-forward resumes immediately instead of waiting out up to 60 seconds of exponential backoff per hop.
• Guard timer-versus-signal races under the FSM lock, queue retry work off the connectivity callback thread, and unsubscribe on executor shutdown.

Why

In the nested E2E failure, a short upstream AMQP disruption becomes a multi-minute store-and-forward delay. Detection is one part of the delay, but the endpoint executor adds another sawtooth: if connectivity returns while the FSM is in Failing, the queued message waits for the already-armed retry timer.

A local reproduction against the real StoringAsyncEndpointExecutor and EndpointExecutorFsm measured:

Outage	Drain after recovery
0s	3.4s
15s	13.9s
60s	1.0s
120s	53.3s

The variance depends on where recovery lands in the 1-to-60-second exponential-backoff cycle. The new recovery signal interrupts that timer. A focused test parks the FSM in a 60-second retry interval and verifies delivery in under 5 seconds after recovery; the real 15-second outage shape also passes the under-5-second assertion.

Validation

• Microsoft.Azure.Devices.Routing.Core.Test: 1,159 passed
• Microsoft.Azure.Devices.Edge.Hub.CloudProxy.Test: 183 passed, 7 skipped
• ConnectionManagerTest: 25 passed
• edgeHub Service build: 0 warnings, 0 errors
• Nested E2E: pending CI image build

Draft scope

The SDK bridge is enabled for the E2E experiment. Make it configurable before merge.

[jlian]

Dug into why the full fix (ce1f114) didn't make RouteMessageL3LeafToL4Module pass on the latest nested-E2E run (172022659, ci-build 20260712.1). Two findings worth capturing here.

1. The fix's retry path never engaged. EndpointExecutorFsm.RetryNowAsync() only acts when state == Failing, but in this repro no endpoint executor is parked in Failing with the message queued at reconnect. Reading the per-test L3 bundle + L4 support bundle:

• On L3, the leaf's message was sent upstream successfully ~89s before the drop; L3's endpoint never entered Failing, so there was nothing to retry.
• On L4, the edgeHub container is delete+recreated by the mid-test deployment reconcile, and its store lives in the ephemeral /tmp/edgeHub, so the queued message is lost with the wiped store rather than parked for a retry. The new L4 relayer1/input1 endpoint shows 0 messages ever added.

So the ConnectivityRecovered -> RequestRetry -> RetryNowAsync chain is inert here: the failing message isn't in a surviving Failing-parked FSM.

2. The test's success log is misleading (unrelated to this fix, but affects how we read runs). IotHub.ReceiveEventsAsync swallows the cancellation on timeout and the caller doesn't assert a match, so the "Received events from module relayer1" line is emitted whenever the receive loop exits, including at the 300s cancellation, regardless of whether a message arrived. Confirmed against the May-29 pass (166045060): a real pass logs "Received events" ~1s after send and the leaf-delete cleanup succeeds; a fail logs it at the 300s cancel instant and then fails on the leaf-delete throwing cancelled. The log line alone does not prove a message arrived.

New angle (experiment running): I opened #7529 (draft) to give the nested-E2E edgeHub a persistent (host-mounted) store so it survives the container recreate, and queued it against a plain main binary (no #7528) to isolate the store-wipe hypothesis. If RouteMessage passes with just the persistent store, the failing message was being lost to the store wipe (a test-rig gap, not the reconnect re-pump). If it still stalls, that cleanly isolates the defect to the store-and-forward re-pump in a surviving store, which is what this PR should target. Will post the run result here.