Investigate hosted smoke failure after config cleanup merge

[jeremi]

Migrated from: jeremi/registry-lab#101
Original author: @jeremi
Source repository: jeremi/registry-lab
Source issue: #101
Source labels: none
Source milestone/release intent: none
Target area/path: lab

Issue

Context

After merging configuration cleanup PR #99, the main push run for hosted-lab failed:

• Run: https://github.com/jeremi/registry-lab/actions/runs/27866519513
• Commit: 02c5c21bfe2885197eb26c6bfffdbec9aa18494e
• Workflow: hosted-lab
  [redacted private/internal deployment detail during migration]
• Failing step: Run hosted smoke

[redacted private/internal deployment detail during migration]

Failure observed

hosted-smoke.py failed with scenario-step-status-mismatch.

The smoke response for NID-1001 against `[redacted non-GitHub deployment URL during migration] reported:

• actual friendly status: needs_attention
• HTTP status fact: 403
• requested evidence: Civil Status Evidence
• answer: Unknown
• raw source rows included: No

Acceptance criteria

• Determine whether this is caused by the config cleanup merge, hosted environment state, credentials/purpose policy, or expected scenario data drift.
• If the smoke expectation is correct, fix the hosted deployment/config/data so the smoke returns the expected scenario status.
• If needs_attention is now the correct behavior, update scripts/hosted-smoke.py or the scenario expectation with a focused test explaining why.
• Re-run the hosted-lab workflow on main and confirm it passes, or document any intentional non-blocking hosted deployment failure mode.

Note: one or more private/internal references were redacted during migration.

Source Discussion

@jeremi commented on 2026-06-20T10:38:01Z

Rerun confirmation: I reran failed jobs on run https://github.com/jeremi/registry-lab/actions/runs/27866519513. Attempt 2 failed the same way.

• Attempt: 2
  [redacted private/internal deployment detail during migration]
• Step: Run hosted smoke
• Failure: scenario-step-status-mismatch
• Actual friendly status: needs_attention
• HTTP status fact: 403
• Subject: NID-1001
• Requested evidence: Civil Status Evidence
• URL in request source: https://shared-notary.lab.registrystack.org/v1/evaluations

The config cleanup merge 02c5c21 touched justfile, scripts/check-stale-config-vocabulary.sh, scripts/doctor-active-configs.sh, scripts/generate-demo-secrets.py, scripts/test_doctor_active_configs.py, scripts/test_lab2_governed_config_reports.py, and tools/lab2-governed-config/src/main.rs. It did not touch the hosted scenario code or the hosted Notary YAML, so this currently looks like hosted deployment/config/data drift rather than a direct code diff in the cleanup PR. Needs investigation before changing expectations.

Note: one or more private/internal references were redacted during migration.

Decision Needed

Hosted smoke failure depends on live environment/rerun evidence and needs Jeremi decision.

Migration Metadata

• Migrated to the public monorepo on 2026-06-25.
• Source title, body, labels, milestone/release intent, and pre-migration discussion were preserved where available.
• Code-grounded audit note: Hosted smoke failure depends on live environment/rerun evidence and needs Jeremi decision.
• Private/internal references, secrets, and obvious deployment-only details were redacted instead of copied forward.