review(hardening): update_branch has no non-retryable failure class and no per-class metric (workflow-permission cause resolved 2026-07-24)

[JSONbored]

Summary

59% of all update_branch failures are a permission error that retrying can never fix, and the engine retries them anyway — up to 9 times on a single PR. The GitHub App cannot push a branch update to any PR that touches .github/workflows/**, but nothing detects this, so every wake re-attempts it.

Production evidence (edge-nl-01, 7 days to 2026-07-27)

agent.action.update_branch: 159 events — 77 completed, 71 error, 11 denied (45% failure rate). The failure breakdown:

Outcome	Failure class	Count	Distinct PRs
error	workflow-permission — unfixable by retry	48	14
error	merge conflict	17	8
denied	head moved (self-race)	6	2
denied	PR no longer open	5	5
error	no new commits on base	5	1
error	GitHub timeout	1	1

The dominant class is:

refusing to allow a GitHub App to create or update workflow '.github/workflows/…'

Worst repeat offenders — attempts against the same PR:

JSONbored/loopover#8499    9
JSONbored/loopover#8497    7
JSONbored/loopover#8400    6
JSONbored/loopover#7736    5
JSONbored/loopover#8030    4

Each attempt is a wasted GitHub API call, an audit row, and a retry cycle, on an operation that is structurally impossible for that PR until its diff stops touching workflow files.

This corrects an earlier characterisation: the 45% failure rate is not primarily the engine racing its own pushes. Self-race accounts for 6 of 82 failures; missing App permission accounts for 48.

Why it matters beyond the waste

• Fourteen PRs — every contributor PR touching CI config — can never satisfy the staleness rebase, so they are permanently "not ready" by that gate's reckoning and re-attempt on every wake.
• It inflates the failure-rate signal, masking the genuine self-race cases (which are real, and are addressed by review: readiness gate force-rebases speculatively — unrelated merges cancel and restart CI on a phase-locked cohort of PRs #9497 and locks: 600s actuation TTL underruns the AI review it wraps since #9013, with no renewal → concurrent double actuation #9467).

Requirements

1. A rebase that cannot succeed for a structural reason must not be attempted repeatedly.
2. A PR touching workflow files must still be reviewable — the inability to rebase it must not become an inability to process it.

Deliverables

• Detect the workflow-permission failure class explicitly and mark it non-retryable for this PR at this diff shape — do not simply catch and continue, or the next wake repeats it.
• Skip the staleness rebase up front when the PR's changed-file set includes .github/workflows/**. The changed-file list is already fetched (pull_request_files), so this is a cheap pre-check rather than a failed API round trip.
• Decide the policy for such PRs and make it explicit: proceed to review un-rebased (the likely right answer given branch protection is strict: false, so being behind blocks nothing), or hold with a named reason. Silently retrying forever is the one option that should be off the table.
• Consider whether the App should hold the workflows permission at all. If it should not, document that so this is understood as intended rather than as a gap.
• Add a metric on update_branch outcome by failure class, so the next regression here is visible without a database query.

Tests (must fail against current main)

• A PR whose diff touches .github/workflows/** does not attempt a staleness rebase.
• A workflow-permission error is recorded once and not re-attempted on the next wake.
• A merge-conflict failure still surfaces as it does today (it is genuinely informative and should not be swept into the same bucket).

Related

• review: readiness gate force-rebases speculatively — unrelated merges cancel and restart CI on a phase-locked cohort of PRs #9497 — the speculative rebase trigger that generates these attempts in the first place. Fixing that reduces the volume here substantially; this issue is still worth fixing independently, because a merge-boundary rebase on a workflow-touching PR would fail the same way.

[linear-code]

JSO-6562

[JSONbored]

---

CORRECTION (2026-07-27, verified) — the trigger is main's workflow changes, not the PR's

The original framing above ("a PR that touches .github/workflows/**") is wrong, and so is the deliverable that followed from it. Verified against the actual PRs:

PR #8499  workflow files in PR = 0   test(scoring): cover model.ts's unpinned-fallback…
PR #8400  workflow files in PR = 0   fix(auth): stop auditing routine device-flow polls
PR #7736  workflow files in PR = 0   fix(engine): normalize repoFullName casing…
PR #8030  workflow files in PR = 0   docs(signals): move orphaned review.enrichment doc
PR #8497  workflow files in PR = 1   fix(ci): stop swallowing control-plane coverage…

Four of the five worst offenders do not touch workflow files at all.

The real mechanism: update-branch merges main into the PR head. main changed .github/workflows/ repeatedly during the affected window (#9245, #9214, #9208, #8624, #8622, #8621, #8620, #8619), so the resulting merge would create/update workflow files in the PR branch — which GitHub refuses for an App without the workflows scope, irrespective of the PR's own diff.

Consequences of the correction

1. Blast radius is far wider than 14 PRs. Every open PR sitting behind a workflow-touching commit on main is un-rebasable for as long as it is behind. Given how often CI config changes here, that is periodically the entire backlog — which also explains why the same PRs are retried so many times (test(scoring): cover model.ts's unpinned-fallback and language-fetch-failed warning branches #8499 nine times): they never stop being behind.
2. The originally-proposed pre-check is ineffective. Checking the PR's own changed files against .github/workflows/** would have caught 1 of these 5. The correct condition is whether the base delta (what the merge would bring in) touches workflow paths.
3. The primary fix is the App permission, not a code workaround. The merge being refused is legitimate.

Revised deliverables

• Grant the loopover-orb App Workflows: Read and write (Repository permissions), and approve the newly-requested permission on the installation. This is the root fix.
• Keep a non-retryable classification for this error class regardless — if the permission is ever absent or revoked, the engine must not retry an impossible operation up to 9 times per PR. Classify and record once.
• Replace the proposed "PR touches workflows" pre-check with the correct condition — the base delta touching .github/workflows/** — or drop the pre-check entirely in favour of the permission fix plus the non-retryable classification. Prefer the latter: once the permission is granted, the pre-check only adds a failure mode of its own.
• Metric on update_branch outcome by failure class still stands.

Note on interaction with #9497

Once #9497 moves the rebase to the merge boundary, this stops being a recurring backlog-wide problem and becomes a rare single-PR one — but it must still be handled, because a merge-boundary rebase on a PR behind a workflow change fails identically.

[JSONbored]

---

SECOND CORRECTION (2026-07-27) — this is resolved; the issue is now historical

The loopover-orb App already holds Workflows: Read and write, and the failures stopped three days ago:

day          workflow_perm_errors   last_seen
2026-07-24            39            2026-07-24T16:51:21.979Z
2026-07-22             4            2026-07-22T14:05:33.791Z
2026-07-21             5            2026-07-21T12:03:36.583Z

That is all 48 errors (39 + 4 + 5), and none have occurred since 2026-07-24 16:51 UTC. Successful update_branch operations have continued normally since — most recent 2026-07-27 15:30 UTC.

The permission was granted/accepted on the afternoon of 07-24, which resolved it. Both prior analyses in this issue — the original "PRs touching workflow files" framing and the first correction's "grant the permission" recommendation — were derived from a 7-day aggregate that straddled the fix, without checking recency. The lesson for future audit work: always bucket a failure population by time before treating it as current.

Revised scope — what is actually left

The root cause is gone. What remains is a small robustness gap, worth keeping but at much lower priority than originally filed:

• The engine retried a structurally impossible operation up to 9 times on a single PR (test(scoring): cover model.ts's unpinned-fallback and language-fetch-failed warning branches #8499) with no notion of "this failure class can never succeed by retrying". If the permission is ever revoked, or another permanent refusal class appears (org policy, ruleset, protected path), the same unbounded retry behaviour returns.
• Add a metric on update_branch outcome by failure class, so a recurrence is visible in Grafana rather than requiring a database query. This is the deliverable that would have surfaced the original problem — and its resolution — without any of the above analysis.

Dropped from scope: the App permission change (already in place) and the changed-file pre-check (would have caught 1 of 5 cases, and is unnecessary now that the permission is correct).

Suggest re-labelling this as a low-priority hardening item, or closing it in favour of the metric deliverable alone.

[JSONbored]

---

THIRD NOTE — the maintainer confirms the permission predates the failures, so the cause is still unexplained

The App has held Workflows: Read and write and the repo installations were approved weeks before these failures. So 2026-07-24 was not a permission fix, and the root cause remains unknown.

What is ruled out:

• Token permission scoping. createInstallationAccessToken (src/github/app.ts:216-227) POSTs to /app/installations/{id}/access_tokens with no body — no permissions subset, no repositories subset. The minted token therefore inherits the installation's full permission set, so a narrowed token is not the explanation.
• "Main stopped touching workflows." Main touched .github/workflows/ on 2026-07-27 (e4a561bee 00:05, 2aa37c35a 02:05, d9bc1ef01 02:25 PT) — after the last error — and an update_branch succeeded at 2026-07-27 15:30 UTC, which would have merged those commits in. So the operation demonstrably works now under the same conditions that previously failed.

Remaining hypotheses, none verified:

1. A cached installation token minted before the permission update, still in the Redis token cache (gh:insttoken) and reused until expiry. This fits a burst that ends abruptly, but not a weeks-old grant — unless the permission was re-accepted or the App re-installed nearer 07-24.
2. A repository ruleset or branch-protection rule (distinct from App permissions) that transiently blocked workflow-file writes, since relaxed.
3. An org-level policy on Actions/workflow modification that changed state.
4. The affected PRs' base deltas genuinely differed in a way not captured by the day-level bucketing above.

Recommended next step

Do not chase this speculatively — it is not currently failing. Instead land the observability deliverable, which makes a recurrence self-diagnosing:

• Metric on update_branch outcome by failure class (permission / conflict / head-moved / closed / other).
• Non-retryable classification for permanent refusal classes, so a recurrence costs one attempt per PR instead of nine.

If it recurs, the metric plus the App's own audit log (Settings → Advanced → Recent deliveries, and the org audit log filtered to integration_installation.*) will identify which of the four hypotheses is correct in minutes, rather than requiring another archaeology pass.