test: persist nested-e2e edgeHub store across container recreate (diagnostic)#7529
Draft
jlian wants to merge 4 commits into
Draft
test: persist nested-e2e edgeHub store across container recreate (diagnostic)#7529jlian wants to merge 4 commits into
jlian wants to merge 4 commits into
Conversation
… L4/L5 support bundles Not for merge. Two temporary changes to investigate the consistently-failing RouteMessageL3LeafToL4Module nested-e2e test (AMQP-only): 1. e2e-run.yaml: narrow the nestededge_amqp filter to the smallest set that reproduces the failure (Module + Metrics + IoTEdgeCheck + RouteMessage), derived from PR #7521's bisection (run 20260701.8 first-failing subset). 2. nested-e2e.yaml: add a Collect_Nested_Bundles job that captures support bundles + raw module logs from ALL levels (L3/L4/L5) after the test stage. The default pipeline only collects the L3 bundle, so the L4 relayer/edgeHub logs that explain the 4-minute store-and-forward stall are never captured.
…dle collection) The temporary category filter narrowed and reordered the nested suite, causing RouteMessage to run first before relayer1 was deployed on L4 (a different failure than the original 15s upstream blip). Revert the filter so the full suite runs with RouteMessage in its normal late position; keep only the Collect_Nested_Bundles job so L4/L5 evidence is captured without perturbing the test flow.
CreateDeviceIdentityAsync now deletes+recreates on DeviceAlreadyExists so an orphaned identity from a prior killed run doesn't block the RouteMessage repro.
…diagnostic) The nested-e2e edgeHub stores store-and-forward messages at the default /tmp/edgeHub, which lives in the container's ephemeral writable layer. When the AMQP-group redeploy recreates the L4 edgeHub container mid-test, any queued/undelivered message is lost with the wiped store. Mount a host directory (/etc/iotedge/storage) and point edgeHub's StorageFolder at it so the message store survives an edgeHub container recreate. edgeHub's entrypoint (hubStart.sh) creates and chowns $StorageFolder/edgeHub to the edgehub uid, so no host-side prep is needed. Diagnostic intent: separate two hypotheses for RouteMessageL3LeafToL4Module. If the test passes with a persistent store, the failing message was being lost to the container-recreate store wipe. If it still stalls, the message is orphaned within a surviving store (the edgeHub store-and-forward re-pump / checkpoint defect that #7528 targets).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The nested-E2E edgeHub stores store-and-forward messages at the default
/tmp/edgeHub, which lives in the container's ephemeral writable layer. When the AMQP-group redeploy recreates the L4 edgeHub container mid-test, any queued or undelivered message is lost with the wiped store.This mounts a host directory (
/etc/iotedge/storage) into edgeHub and points itsStorageFolderat it, so the message store survives an edgeHub container recreate. edgeHub's entrypoint (hubStart.sh) creates and chowns$StorageFolder/edgeHubto the edgehub uid at startup, so no host-side preparation is needed.Applies to both middle-layer AMQP deployments (
nestededge_middleLayerBaseDeployment_amqp.jsonandnestededge_middleLayer_e2e_amqp.json).Why (diagnostic)
Investigating why
RouteMessageL3LeafToL4Modulefails, the message the test waits on is stranded when the L4 edgeHub container is delete+recreated by the mid-test deployment reconcile. Because the store is ephemeral, the queued message is destroyed with the old container rather than being retried on reconnect.This change is a diagnostic to separate two hypotheses:
Draft scope
Draft, not for merge as-is. This is a targeted experiment to inform the durable edgeHub store-and-forward fix. If persistent storage is the right long-term posture for the nested-E2E rig (arguably it is, since store-and-forward across a restart is a real customer scenario), the mount path and provisioning should be reviewed for the CI agents.