Skip to content

Adaptive implementer tier on fix passes; pin ponytail-review to sonnet#14

Open
jirispilka wants to merge 1 commit into
claude/shepherd-engineering-practices-2chpb1from
claude/shepherd-adaptive-tiering
Open

Adaptive implementer tier on fix passes; pin ponytail-review to sonnet#14
jirispilka wants to merge 1 commit into
claude/shepherd-engineering-practices-2chpb1from
claude/shepherd-adaptive-tiering

Conversation

@jirispilka

Copy link
Copy Markdown
Collaborator

What we're solving

Follow-up iterations solve progressively smaller findings, but model tiering priced every iteration like the first — and auto was resolving ponytail-review, a one-line deletion/leanness lens, to opus on medium/large runs.

The changes are deliberately asymmetric because the loop's economics are asymmetric: under zero-findings convergence, a reviewer's hallucinated finding costs a full fix round (implementer + oracle + full re-review), and cheaper judges raise the false-finding rate — so implementation may get cheaper across iterations while judges must not.

How

  • Fix pass (mirrors the architect's existing revision-pass downgrade): on iteration ≥2 with only minor/nit findings open and no design-level change, an auto-resolved implementer drops one tier (sonnethaiku), bumping back if a finding survives its round. Explicitly pinned models never drop. Reviewer models never change mid-loop — the PASS that ends the loop must not come from a weaker judge than the FAIL that opened it (verdict continuity).
  • Default config pins ponytail-review to sonnet: its findings are minor/nit by construction, so opus buys nothing, while sonnet keeps the review floor.
  • docs/shepherd-config.md updated to match; contract + config tests pin both rules.

94 tests pass; SKILL.md at 448/450 lines.

Alternatives considered

  • Downgrading reviewers on later iterations — rejected: breaks verdict continuity (a weaker judge's PASS is indistinguishable from "doesn't see the findings") and false positives from cheap judges are the loop's dominant cost, not judge tokens (Copilot threads in crawlee-python #1551 ran ~50% false positives).
  • haiku for ponytail — rejected: the review floor stays sonnet; a hallucinated "shrink" nit forces a full fix round, and ponytail's clarity guard is exactly the nuance weak models drop.
  • Schedule-free alternative for panel pruning: /shepherd-evaluate overlap data should drive per-repo roster cuts (a reviewer that keeps returning zero findings is a cut candidate) — bigger savings than any model downgrade, left to run evidence rather than encoded upfront.

Notes

Stacked on #13 — merge that first; this PR's base is that branch, so the diff shows only this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_016G4uCkx2btpqfGtJRAPJKd


Generated by Claude Code

Follow-up iterations solve progressively smaller findings, but model
tiering priced every iteration like the first. Two changes, asymmetric
by design because loop economics are asymmetric — under zero-findings
convergence a reviewer's hallucinated finding costs a full fix round,
so judges must not get cheaper while implementation may:

- Fix pass: on iteration >=2 with only minor/nit findings open and no
  design-level change, an auto-resolved implementer drops one tier
  (sonnet -> haiku), bumping back if a finding survives its round.
  Explicitly pinned models never drop. Reviewer models never change
  mid-loop: the PASS that ends the loop must not come from a weaker
  judge than the FAIL that opened it. Mirrors the architect's existing
  revision-pass downgrade.
- Default config pins ponytail-review to sonnet instead of auto (which
  resolved to opus on medium/large): a one-line deletion/leanness lens
  doesn't earn opus, and sonnet keeps the review floor.

docs/shepherd-config.md updated to match; contract + config tests pin
both rules; 94 tests pass; SKILL.md at 448/450 lines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016G4uCkx2btpqfGtJRAPJKd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants