AI review intermittently publishes no narrative summary (~40% of reviews) — root cause unknown

[JSONbored]

Summary

Some AI reviews publish a real narrative summary; others publish the generic placeholder ("The AI review returned non-blocking notes for this change but did not include a separate narrative summary") or "AI review is unavailable for this PR head" — under what appears to be identical configuration. Root cause is not established.

Evidence (2026-07-27, all after AI_MAX_OUTPUT_TOKENS was restored to 4096)

PR	cached summary state	time (UTC)
loopover#9426	REAL_SUMMARY	18:21
awesome-claude#5599	UNAVAILABLE	18:20
loopover#9423	REAL_SUMMARY	18:03
loopover#9417	FALLBACK_PLACEHOLDER	18:01
loopover#9420	REAL_SUMMARY	17:58
loopover#9415	REAL_SUMMARY	17:56
loopover#9422	FALLBACK_PLACEHOLDER	17:55
loopover#9421	FALLBACK_PLACEHOLDER	17:49

Roughly 40% degraded. Query used:

SELECT repo_full_name, pull_number, created_at,
  CASE WHEN notes LIKE 'AI review is unavailable%' THEN 'UNAVAILABLE'
       WHEN notes LIKE 'The AI review returned%did not include a separate narrative%' THEN 'FALLBACK_PLACEHOLDER'
       ELSE 'REAL_SUMMARY' END AS summary_state
FROM ai_review_cache WHERE created_at > '2026-07-27T17:37:00Z' ORDER BY created_at DESC;

Ruled out

• Provider auth — claude CLI in the container works when invoked as the app does (CLAUDE_CODE_OAUTH_TOKEN sourced from /run/secrets/claude_code_oauth_token → returns OK). A bare docker exec claude -p ... reports "Not logged in", but that is a test artifact: loadFileSecrets populates process.env inside the Node process at boot and docker exec gets a fresh shell.
• Output token budget — AI_MAX_OUTPUT_TOKENS was transiently set to 1024 (see #ISSUE2), which throttled ai-review.ts's own 4096 default and caused degraded reviews between ~16:24–17:36 UTC. Restored to 4096; degradation persists for a subset afterwards, so this is not the whole story.
• Stale cache replay — the re-run checkbox sets forceAiReview: true, and processors.ts explicitly bypasses the cache (webhook.forceAiReview === true ? null : getCachedAiReview(...)). Re-runs do make a fresh LLM call.
• Bare-score public-safe filter — a real contributing cause, now fixed separately (see below), but it does not explain PRs that succeed and fail in the same window.

Suspected but unverified

composeAdvisoryNotes reaching the UNAVAILABLE branch means it returned null and composeFallbackAdvisoryNotes returned null — i.e. no assessment, no blockers, no nits parsed at all. For awesome-claude#5599 this reproduced across two consecutive forced re-runs while ai_usage_events recorded status=ok, output_tokens 413–442. So the provider returned output that failed to parse into the expected structure. Worth checking whether parseModelReview is silently dropping a valid-but-differently-shaped response, and whether small PRs elicit a response shape the parser does not handle.

Why this matters

gate.aiReview.mode: block is set for these repos. A review with no usable assessment routes to a manual-review hold, so this directly stalls the merge pipeline.

Related

• Public-score-terms allowlist was unset in production, which silently discarded whole narrative summaries for any review using the word "score"/"scoring". Fixed by setting LOOPOVER_PUBLIC_SCORE_TERMS_ALLOWED_REPOS and by the preflight in the linked PR.
• The no-narrative placeholder was being persisted into ai_review_cache.notes and read back as a genuine assessment. Fixed in the linked PR.

[linear-code]

JSO-6520

[JSONbored]

Correction — one theory in the original description is wrong, do not act on it.

The description notes that the no-narrative placeholder is persisted into ai_review_cache.notes and reads back as a genuine assessment. That part is factually true, but treating it as a bug is not correct, and a fix along those lines was written and then reverted.

Making hasPublicReviewAssessment reject the persisted placeholder causes the orchestration to take the ai_review_public_summary_missing branch instead, which:

• holds the PR for manual review rather than publishing, and
• discards the model's nits entirely from the panel.

Two existing tests pin the intended behaviour and correctly failed:

• ai-review-advisory.test.ts > preserves model nits when the model omits the assessment summary
• queue-2.test.ts > publishes AI notes when the review omits a narrative assessment

So "placeholder + nits published" is deliberate product behaviour — publishing real findings beats holding on a missing narrative. The persisted placeholder is a consequence of that design, not corruption.

This narrows the actual investigation. The problem is not that the fallback exists or is cached; it is that the fallback fires too often. Focus remains on why assessment comes back empty — i.e. either the model genuinely omits a narrative, or parseModelReview / the public-safe filter drops it. The bare-score filter was one confirmed cause of the latter (now mitigated); the residual ~40% is still unexplained.

Only the boot-time allowlist warning from that work was kept (see the linked PR).

[JSONbored]

Concrete next step: the answer is already being recorded, just not where it can be read.

AiReviewDiagnostic.status (src/services/ai-review.ts) already distinguishes precisely the cases this issue is trying to separate:

status: "parsed" | "empty_output" | "unparseable_output" | "provider_error" | "missing_assessment" | "identical_retry_skipped"
responseChars?: number
hasJsonObject?: boolean

That is exactly the discriminator needed: missing_assessment would mean the model genuinely returned no narrative, whereas unparseable_output / empty_output (with hasJsonObject: false) would mean the response never parsed at all.

Mechanism that produces the UNAVAILABLE outcome: parseModelReview returns null unless extractLastJsonObject finds a balanced {...}. Prose-only output, or output truncated mid-JSON, yields null → no assessment, no blockers, no nits → composeAdvisoryNotes returns null → the ai_review_public_summary_missing branch. This fits awesome-claude#5599, which reproduced across two consecutive forced re-runs with status=ok and only 413–442 output tokens.

Observability gap blocking the diagnosis: these diagnostics are emitted only through capturePostHogReviewFailure (as review_diagnostics). They do not appear in the container logs / Loki — a docker logs grep over the incident window for unparseable_output|empty_output|missing_assessment|ai_review_public_summary_missing returns zero matches. So the data needed to close this issue exists but is reachable only via PostHog.

This mirrors a pattern the codebase already calls out elsewhere (e.g. the orb_export_failed catch that "was silent beyond the counter ... never sees it either"). Suggest emitting the same structured diagnostic to the console alongside the PostHog capture, so this is debuggable from Loki without a third-party dashboard.

Suggested order of work:

1. Add a structured console log for review diagnostics (cheap, unblocks everything below).
2. Read status + hasJsonObject for a degraded review and confirm which branch fires.
3. If unparseable_output dominates: inspect the raw provider response — in particular whether the fallback reviewer (ollama:qwen3:8b, a reasoning model that emits a separate reasoning field) is answering and returning non-JSON or truncated JSON, since the reviewer plan is claude-code:claude-sonnet-5->ollama:qwen3:8b.

[JSONbored]

Update: FALLBACK_PLACEHOLDER class root-caused and fixed; UNAVAILABLE class still open

Split into the two distinct failure classes this issue's evidence table already showed, since they turned out to have different causes.

FALLBACK_PLACEHOLDER (loopover#9417, #9422, #9421) — fixed

Root cause was not the bare-score pattern (that was a real but separate contributing cause, already noted above). It was FORBIDDEN_PUBLIC_COMMENT_WORDS itself: sanitizePublicComment matches a list of explicit-phrase terms (reward, ranking, cohort, farming, reviewability, ...) via plain case-insensitive .includes(), and toPublicSafe discarded the model's entire narrative assessment the instant any one sentence used any of these — which is ordinary English review vocabulary, not a private-value leak. A review of this codebase's own gate/scoring code (exactly the review target here) inevitably uses this vocabulary.

Confirmed empirically: 3 of 4 realistic review sentences I tested were rejected wholesale by the old all-or-nothing check. metadata_json.inconclusive on the affected cache rows was false (the review had genuinely succeeded — this wasn't a provider failure), consistent with the assessment existing and being discarded downstream, not missing.

Fixed by filtering per sentence instead of all-or-nothing (toPublicSafeBySentence, merged in #9445) — matching how nits/blockers were already filtered per-item. A sentence naming a real private value is still dropped in full (verified: the only sentence-boundary-straddling risk would be a forbidden entry itself containing a ./!/?, and none do); a sentence that merely discusses this repo's own public vocabulary now survives.

UNAVAILABLE (awesome-claude#5599) — still open, but now diagnosable

The suspected mechanism (parseModelReview returning null on a response shape it can't parse) is still unverified — I could not confirm it this session because the diagnostics that would answer it only ever reached PostHog, and this session had no PostHog auth and no running docker logs to check container output either.

#9447 (merged) persists the per-attempt diagnostic (missing_assessment / unparseable_output / empty_output / provider_error) into ai_review_cache.metadata_json on every degraded review going forward, written only on the degraded path (not on every review, to avoid growing the database — see #9435's history). The next time this reproduces, the cause is one query away instead of requiring PostHog access.

Leaving this open, now scoped to just the UNAVAILABLE class.