diff --git a/.gitignore b/.gitignore index 436344f..1e51ce1 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,6 @@ # Local agent config (not shared) .claude/settings.local.json + +# skill-creator eval scratch (not part of the skill) +*-workspace/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..66c7604 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,32 @@ +# CLAUDE.md — reply-pr + +Guidance for Claude Code when working in this repo. + +## What this repo is + +`reply-pr` is a cross-tool **Agent Skill** (`SKILL.md`) that triages and replies to +a pull request's unresolved review comments (reply-only; it never edits code). The +skill lives at **`skills/reply-pr/`** (`SKILL.md` + `scripts/`); the repo root holds +the README and config. It installs via `npx skills add deployhq/reply-pr` — which +requires the `skills//SKILL.md` layout, so keep that structure. + +## Repo hygiene — do NOT commit scratch or generated artifacts + +Only the skill, README, and repo config belong in git. Never commit local scratch, +eval output, or tool-generated working directories: + +- `*-workspace/` (e.g. skill-creator's `reply-pr-workspace/` eval runs), `.idea/`, + `.vscode/`, `.claude/settings.local.json` — all gitignored; keep it that way. +- **Do not run `git add -A` / `git add .` blindly** — it sweeps in whatever untracked + scratch happens to be in the working tree. Stage specific paths (`git add `), + or run `git status` and confirm exactly what's staged **before** committing. + +Why this rule exists: an earlier `git add -A` accidentally committed a skill-creator +eval workspace into this public repo. Verifying the staged set prevents persisting +things we don't need. + +## Testing the skill + +- `bash skills/reply-pr/scripts/gather.sh [owner/repo]` — read-only; safe to run. +- `scripts/post-reply.sh` **posts and resolves** on real PRs — don't run it against a + live PR while testing. diff --git a/README.md b/README.md index ca39ce3..52a4e8d 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,50 @@ -# reply-pr - -**Answer a PR's review comments the right way — without re-explaining the etiquette every time.** - -Reviewers — Codex, Claude, CodeRabbit, and humans — pile comments onto a pull -request, and replying *well* is fiddly: each reviewer wants a different channel -and a different `@`-ping, every comment deserves a reply (what you **tackled** -*and* what you **skipped**, with a reason), pre-existing code shouldn't be -re-litigated, and you're meant to end with **zero unresolved threads**. Doing all -that by hand is tedious and easy to get subtly wrong. - -`reply-pr` turns a wall of review comments into a clean, decided set of replies. -It knows the per-reviewer conventions, critically assesses each comment (it does -**not** blindly accept bot suggestions), and **never posts without your OK**. - -## Why use it - -- **Stop re-explaining the rules.** It already knows them: Codex → inline + `@codex`, - Claude → top-level + `@claude`, CodeRabbit → inline, no ping, nitpicks dropped, - humans first. Any other bot works too — it replies *in kind*. -- **You stay in control.** It is **reply-only — it never edits code**, and it - **always stops at a deliberation gate** with a tackle/skip proposal before - posting anything. -- **No silent skips.** Every comment gets a reply — what you did (with the commit - SHA) or why you skipped it — in your voice. -- **Cuts the noise.** Drops CodeRabbit nitpicks and flags findings that look - pre-existing (a bot re-flagging code this PR didn't change). -- **Ends clean.** Resolves threads once they're settled, so the PR lands at zero - unresolved. -- **Cross-tool.** Runs in Claude Code, Cursor, Codex — any tool that reads Agent - Skills. +

reply-pr

+ +

+ Triage and reply to a pull request's unresolved review comments —
+ reply-only, on each reviewer's channel, with you approving every reply. +

+ +

+ License: MIT + Agent Skill + Works with Claude Code, Cursor, Codex +

+ +`reply-pr` reads the unresolved comments on a pull request, helps you decide what +to address and what to skip, and posts a reply on the right channel for each +reviewer — in your voice. It doesn't edit your code, and it doesn't post anything +you haven't approved. + +## Why it exists + +A PR usually gathers comments from several reviewers at once — CodeRabbit, Codex, +Claude, and people. Replying to them properly is more fiddly than it looks: + +- each reviewer expects a different channel and `@`-ping (Codex inline with + `@codex`, Claude a top-level comment with `@claude`, CodeRabbit inline, no ping); +- every comment wants a reply — including the ones you skip, with a reason; +- bots re-flag pre-existing code and add nitpicks that aren't worth re-litigating; +- and you're meant to leave no unresolved threads behind. + +`reply-pr` keeps track of all that, so the replies stay yours and nothing slips. ## How it works ```mermaid flowchart TD A["reply-pr on your PR"] --> B["Gather UNRESOLVED comments
bots + humans, inline + top-level"] - B --> C["Classify + critically assess
humans first, dedup,
drop CodeRabbit nitpicks,
flag likely pre-existing"] - C --> G{"Deliberation gate:
tackle / skip + rationale
you decide"} + B --> C["Classify + assess
humans first, dedup,
drop CodeRabbit nitpicks,
flag likely pre-existing"] + C --> G{"You decide:
tackle / skip + rationale"} G -->|you steer| R["Reply on each reviewer's channel
item by item, in your voice"] - R --> Z["Resolve settled threads
zero unresolved"] + R --> Z["Resolve settled threads"] ``` -Nothing is posted, no code is touched, and no thread is resolved **before the -gate** — the skill only acts on what you approve. +It gathers the unresolved comments, assesses each (humans first, CodeRabbit +nitpicks dropped, likely-pre-existing findings flagged), and shows you a +tackle/skip list to steer. Once you approve, it posts each reply on the reviewer's +channel — noting the commit that fixed things where relevant — and resolves the +settled threads. It is reply-only, and it never posts before you approve. ## Install @@ -50,34 +52,31 @@ With the [`skills`](https://github.com/vercel-labs/skills) CLI — cross-tool (C Code, Cursor, Codex, …): ```bash -npx skills add deployhq/reply-pr # this project only (./.claude/skills) -npx skills add deployhq/reply-pr -g # all your projects (~/.claude/skills) +npx skills add deployhq/reply-pr -g # all your repos (~/.claude/skills) +npx skills add deployhq/reply-pr # this project only (./.claude/skills) ``` -`skills add` **defaults to project scope** (committed with the repo, shared with -your team). `reply-pr` is a personal, cross-repo tool, so you probably want -**`-g` / `--global`** instead. Scope is set by the flag — the CLI doesn't prompt -for it. - -Inside saas.group it's also listed on [lookmi](https://lookmi.saas.group/skills) — -install it with the command shown on its skill page. +`skills add` defaults to project scope; since `reply-pr` is a personal, cross-repo +tool, you probably want **`-g` / `--global`**. (Scope is set by the flag — the CLI +doesn't prompt for it.) Inside saas.group it's also on +[lookmi](https://lookmi.saas.group/skills). ## Use -``` -/reply-pr # unresolved comments on the current branch's open PR +```bash +/reply-pr # the current branch's open PR /reply-pr 1016 # a specific PR /reply-pr analyse # analysis only — never posts ``` -Or just ask naturally: *"reply to the review comments on this PR"* — the skill -triggers from its description. Either way it stops at the **deliberation gate** -(a tackle/skip proposal, humans first) and only posts after you steer it. +Or just say *"reply to the review comments on this PR."* Either way it stops at the +approval gate — a tackle/skip proposal, humans first — and only posts what you +approve. -## Reviewer routing +## Who gets replied to, and how -The general rule works for **any** reviewer: reply *in kind* (where they -commented) and ping only reviewers known to want it. +The rule works for any reviewer: reply in kind (where they commented) and ping only +reviewers known to want it. ```mermaid flowchart LR @@ -88,26 +87,27 @@ flowchart LR O["Any other bot"] --> OR["reply in kind, no ping
flagged for you to confirm"] ``` -## What it will and won't do +## What it does and doesn't -- **Will:** fetch unresolved comments, propose tackle/skip (you decide), post the - replies you approve on the right channel, and resolve settled threads. -- **Won't:** edit code, post anything before you approve it, or quietly skip a - comment or re-litigate pre-existing code without flagging it. +| Does | Doesn't | +|---|---| +| Fetch unresolved comments | Edit your code | +| Propose tackle / skip (you decide) | Post anything before you approve it | +| Post replies on the right channel, in your voice | Silently skip a comment | +| Resolve settled threads | Re-litigate pre-existing code without flagging it | ## Requirements -- `gh` CLI (authenticated), `jq`. +- [`gh`](https://cli.github.com/) CLI (authenticated), `jq`. -## Structure +## Under the hood -``` +```text skills/reply-pr/ SKILL.md — the skill (workflow, reply rules, voice) scripts/gather.sh — fetch unresolved threads (GraphQL) + top-level comments scripts/post-reply.sh — post inline / top-level replies, resolve threads -README.md ``` -(The `skills/reply-pr/` layout is what the `skills` CLI expects when installing -from a repo.) +The scripts are deterministic plumbing (`gh` + `jq`); the judgement — what to +tackle, how to word it, when to resolve — stays in the skill, gated on your call. diff --git a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_1.json b/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_1.json deleted file mode 100644 index a317bb1..0000000 --- a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_1.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "iteration": 1, - "prompt": "You are optimizing a skill description for a Claude Code skill called \"reply-pr\". A \"skill\" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.\n\nThe description appears in Claude's \"available_skills\" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.\n\nHere's the current description:\n\n\"Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.\"\n\n\nCurrent scores (Train: 6/12):\n\nFAILED TO TRIGGER (should have triggered but didn't):\n - \"triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?\" (triggered 0/3 times)\n - \"i already pushed the fixes for the review comments, now help me post replies referencing the commits\" (triggered 0/3 times)\n - \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3 times)\n - \"the claude bot reviewed my PR and left notes, help me draft replies and ping it back\" (triggered 0/3 times)\n - \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly\" (triggered 0/3 times)\n - \"reply to the PR comments\" (triggered 0/3 times)\n\nPREVIOUS ATTEMPTS (do NOT repeat these \u2014 try something structurally different):\n\n\nDescription: \"Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.\"\nTrain results:\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\n\nSkill content (for context on what the skill does):\n\n---\nname: reply-pr\ndescription: >-\n Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots\n (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human\n deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the\n user wants to respond to, address, answer, or \"reply to\" PR review feedback;\n handle Codex/Claude/CodeRabbit comments; go through review comments one by one;\n or clear/resolve unresolved review threads \u2014 even if they don't name this skill\n and even if they just say \"let's reply to the PR comments\" or \"check the review\n comments on PR 1016\". Do NOT use it to actually fix the code the reviewers\n flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts\n the replies afterwards.\n---\n\n# reply-pr\n\nReviewers (bots and humans) leave comments on a PR. Answering them *well* means:\nreply on the **right channel for each reviewer**, go **item by item**, give a\n**rationale for both what you tackled and what you skipped**, and leave **no\nunresolved thread behind**. Doing that by hand every time is tedious and easy to\nget subtly wrong (wrong channel, missing ping, silent skips). This skill makes\nthe correct etiquette the default.\n\n## Two hard rules\n\n1. **Reply-only. Never edit code.** Any code fixes were already made in the\n normal dev flow; the replies just reference those commits. If the user hasn't\n made a fix yet, note it as a \"to fix\" in the gate \u2014 don't fix it here.\n2. **Never post without the deliberation gate.** Always stop, show the user your\n tackle/skip proposal, and wait for their steer before posting anything. This\n is the single most important behaviour \u2014 the user has repeatedly said \"do not\n action, just critically check\" / \"do not act, let me know first\".\n\n## The workflow \u2014 three phases\n\n### Phase 1 \u2014 Analyse (read-only)\n\n1. **Find the PR and know your role.** Use the argument if given (`reply-pr 1016`),\n else the open PR on the current branch. Get `owner/repo` and the PR number.\n `gather.sh` returns `self` (the login you post as) and `pr_author` (the PR's\n author). The **common case is `self == pr_author`** \u2014 you're the author\n answering reviewers; treat your own (`self`) comments as prior replies and skip\n them. But **don't assume it**: if `self != pr_author`, you're running as someone\n else (a reviewer or collaborator), so your own comments may be review *findings*,\n not replies \u2014 say so at the gate (\"running as ``, who isn't the author\n ``\") and confirm intent (replying on the author's behalf vs.\n responding to threads as a reviewer), then adapt. The user can also state their\n role in the prompt (e.g. \"I'm reviewing this, not the author\").\n2. **Gather the unresolved feedback.** Run the bundled script:\n `bash scripts/gather.sh [owner/repo]`\n It **fails loudly** on any fetch error (never a false \"no comments\"), paginates,\n and returns JSON with:\n - `self` \u2014 the login you post as (so you can recognise your own past replies) \u2014\n and `pr_author` \u2014 the PR's author (compare the two to know your role; step 1).\n - `unresolved_threads` \u2014 inline review threads with `isResolved == false`\n (filtered client-side; GraphQL has no server-side `isResolved` argument).\n Each has `thread_id` (for resolving), `comment_id` (the **root** comment, for\n `in_reply_to`), `reviewer` (bare login) + `is_bot`, `path`, `line`, `url`,\n `body` (the finding),\n `is_outdated`, **`on_changed_line`** (did this PR add/remove the flagged line,\n or is it unchanged context \u2014 a pre-existing hint; see step 4), and the\n thread's recent history: `comment_count`, `last_comment_author`,\n `last_comment_body`.\n - `top_level_comments` \u2014 PR conversation comments (where Claude-bot posts its\n whole review).\n Read `scripts/gather.sh` to see or adjust the queries.\n3. **Classify + dedup.** For each item, bucket it by reviewer and channel:\n - **Reviewer \u2192 bucket by `is_bot` + login.** `is_bot: false` is a **human**\n (highest priority). For bots (`is_bot: true`), match the login (already\n normalised \u2014 the `[bot]` suffix is stripped): `chatgpt-codex-connector` =\n Codex, `claude` = Claude, `coderabbitai` = CodeRabbit \u2014 use the matrix below.\n An **unknown bot** \u2192 default to an inline reply with **no ping** and flag it\n in the gate for the user to confirm the channel; never silently treat a bot\n as a human. (Classify by `is_bot`, not the login suffix: GitHub's GraphQL\n drops the `[bot]` suffix while REST keeps it, so the suffix is unreliable.)\n - **Dedup/group across reviewers** \u2014 multiple bots often flag the same thing.\n Present it once, noting who raised it.\n - **Drop CodeRabbit *nitpicks*.** Ignore anything CodeRabbit itself badges as a\n nitpick / trivial / minor. But a **substantive bug** CodeRabbit flags is NOT\n a nitpick \u2014 keep it.\n - **Skip what's already handled** (this is the only thing stopping a re-reply\n loop, since top-level comments have no \"resolved\" state). A thread whose\n `last_comment_author` is `self` is one we've already answered \u2014 don't\n re-reply. Among `top_level_comments`, ignore any authored by `self` (our own\n past replies), and skip a bot's review that a later comment already answers.\n4. **Critically assess each \u2014 don't take it at face value.** Verify the claim\n against the actual code/diff. Reviewers (especially bots) are often wrong,\n out of date, or flagging pre-existing code. Decide a **proposed stance**:\n **tackle** or **skip**, with a one-line rationale.\n - **Spotting pre-existing code (two signals \u2014 both advisory; you confirm at the\n gate, they never auto-skip):**\n - `on_changed_line: false` means the bot commented on an **unchanged context\n line** \u2014 code this PR did not touch. Strong \"pre-existing\" hint: lean toward\n skip-as-pre-existing (reply it's not part of this PR's work). `true`/`null`\n \u2192 treat as this PR's code and engage normally.\n - A finding can be a **source-PR duplicate** even on an added line \u2014 the PR\n re-includes code already reviewed in another (merged) PR, so it shows as\n *added* here and `on_changed_line` won't catch it. If you suspect it,\n confirm cheaply: find the commit behind the line (`git blame`, or the\n comment's commit) and run `gh api repos///commits//pulls`\n \u2014 if it returns another PR, the code came from there. Reply that it's from\n PR #X (link the originating thread) and skip.\n - Common skip rationales: *pre-existing / unrelated to this diff*, *out of scope\n / scope creep*, *accepted trade-off (flagged in the PR description for\n sign-off)*, or **already-shipped** (either case above \u2014 reply pointing at the\n origin, don't change anything here).\n\n### Phase 2 \u2014 Deliberation gate (STOP here)\n\nPresent a single ranked list for the user to steer. Order it **humans first**\n(their feedback outranks the bots), then Codex/Claude, then CodeRabbit. For each\nitem show: reviewer \u00b7 channel \u00b7 a one-line summary \u00b7 your proposed **tackle/skip**\n\u00b7 the rationale. Tag anything that looks **pre-existing** (`on_changed_line:\nfalse`, or a suspected source-PR duplicate) with a \"likely pre-existing\" marker so\nit stands out at a glance. Then **stop and wait.** Do not post anything yet.\n\nPhrase it as a proposal they edit, e.g. \"here's each unresolved comment with what\nI'd tackle vs skip and why \u2014 tell me what to change.\" If the user gave a mode\n(`reply-pr analyse`), stop here permanently and don't offer to post.\n\n### Phase 3 \u2014 Reply (only after the user approves)\n\nFor each item the user confirmed, post on the correct channel (matrix below),\n**item by item**, with the rationale \u2014 for **both** tackled and skipped items. A\nsilent skip is never acceptable; if we're not doing something, say why. For a\ntackled item, name the **commit SHA** and what changed.\n\n**Write each reply to a file first, then pass the file** to the helper \u2014\n`post-reply.sh` reads the body from a file **path** (that's what keeps backticks,\nquotes and newlines intact); passing the reply text directly will fail with \"No\nsuch file\". For example: write the reply to `/tmp/reply-.md`, then\n`bash scripts/post-reply.sh inline /tmp/reply-.md` (or\n`toplevel /tmp/reply-claude.md`). See the script header for all args.\n\nThen **resolve threads** (see \"Leaving no thread behind\" below).\n\n## How to reply to each reviewer\n\n**The general rule works for ANY reviewer** \u2014 known bot, unknown bot, or human.\nDon't hardcode to a fixed list; reply *in kind* and ping only when it helps:\n\n1. **Channel = reply where they commented.** An inline review comment \u2192 a threaded\n **inline** reply. A top-level PR comment \u2192 a **top-level** comment. `gather.sh`\n already splits these (`unresolved_threads` = inline, `top_level_comments` =\n top-level), so the right channel falls out for every reviewer automatically.\n2. **Ping = only reviewers known to want re-notifying** (default: **no ping**).\n Pinging re-triggers a bot to re-review \u2014 some want it, some don't, so when in\n doubt, don't.\n3. **Humans first** \u2014 a human's feedback outranks the bots; address it first.\n\n**Known-reviewer specifics.** Anything not listed falls back to the general rule\n(reply in kind, no ping) \u2014 and flag it in the gate so the user can confirm the\nchannel/ping for a bot you haven't seen before:\n\n| Reviewer (`is_bot` + login) | Channel | Ping |\n|---|---|---|\n| **Human** (`is_bot: false`) | in kind (inline or top-level) \u2014 **highest priority** | optional @-mention |\n| **Codex** (`chatgpt-codex-connector`) | inline, threaded under the finding | **`@codex`** |\n| **Claude** (`claude`) | **top-level** comment (it never comments inline) | **`@claude`** \u2014 reference the `#issuecomment-` URL, then go point by point |\n| **CodeRabbit** (`coderabbitai`) | inline (substantive items only; nitpicks dropped) | none |\n| **Any other bot** | in kind (wherever it commented) | none \u2014 flag in the gate to confirm |\n\n**Why Claude is the exception:** it posts its whole review as one top-level comment\nwith no inline threads, so a top-level `@claude` reply referencing the comment is\nthe only thing that lands. Every other reviewer (bots and humans) attaches findings\nto code lines, so threaded inline replies keep the conversation where they put it \u2014\nwhich is exactly what \"reply where they commented\" gives you for free.\n\n## Reply voice\n\nMatch the user's style (from replies he's posted verbatim). Substantive,\ntechnical, concise-but-complete. **No emojis.**\n\n- **Bold per-item headers with the verdict IN the header**, e.g.\n `**1. Nuxt tier-1 observation \u2014 skipped (intentionally, with rationale).**`\n- **Openers:** inline \u2192 `**Addressed in \\`\\`.**` ; top-level (Claude) \u2192\n `**@claude \u2014 this is a reply to your review at .** Going through your points one by one, with what we did and why.`\n- **Verdict vocabulary:** \"acknowledged, no code change\" \u00b7 \"agreed, no code\n change\" \u00b7 \"addressed\" \u00b7 \"correct observation, and we deliberately left it as-is\".\n- **Skip rationale leans on:** \"pre-existing \u2026 unrelated to this diff\" \u00b7 \"folding\n it into this PR would be scope creep\" \u00b7 \"out of scope\" \u00b7 \"flagged in the PR\n description for conscious sign-off\".\n- **Tackled:** name the SHA and what changed (\"Added a spec for a\n `.deployignore`-sourced culprit\").\n- **Closers:** a brief thank-you \u2014 \"Thanks for the catch.\"\n\n## Leaving no thread behind\n\nThe end state the user wants is **zero unresolved/pending threads**. Use each\nthread's `last_comment_author` / `last_comment_body` (from `gather.sh`) to decide\nwithout re-fetching:\n\n- After replying to an inline thread where **our reply is the last word** (a skip\n we've explained, or a fix we've described and nothing more is expected), resolve\n it: `bash scripts/post-reply.sh resolve `.\n- If `last_comment_author` is the **reviewer** (they replied after us \u2014 common\n with Codex, which often confirms), read `last_comment_body`: if it's a\n confirmation, resolve; if they raised a new point, handle it. If the thread is\n still **awaiting the reviewer's follow-up**, **do not resolve yet** \u2014 note it as\n \"waiting on \" and settle it on a later run. When in doubt, leave it\n open and flag it rather than resolving prematurely.\n- If a thread is `is_outdated: true`, the code moved since the comment was made,\n so the point may no longer apply \u2014 check whether it's still valid, say so in the\n reply if it's moot, and it's usually safe to resolve.\n\n## Scripts\n\n- `scripts/gather.sh [owner/repo]` \u2014 fetch unresolved review threads\n (paginated GraphQL, filtered client-side) + top-level comments as classified\n JSON, including `self` and each thread's recent history. Fails loudly on any\n fetch error rather than returning an empty result. Read it if you need to tweak\n the queries or bot-author filters.\n- `scripts/post-reply.sh ...` \u2014 post an inline reply (`inline `),\n a top-level comment (`toplevel `), or resolve a thread\n (`resolve `). It builds the JSON safely with `jq` and uses `gh api`.\n\nBoth require an authenticated `gh` CLI. The scripts are the deterministic\nplumbing; the judgement (classify, assess, gate, voice) stays here in the skill.\n\n\n\nBased on the failures, write a new and improved description that is more likely to trigger correctly. When I say \"based on the failures\", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:\n\n1. Avoid overfitting\n2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.\n\nConcretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters \u2014 descriptions over that will be truncated, so stay comfortably under it.\n\nHere are some tips that we've found to work well in writing these descriptions:\n- The skill should be phrased in the imperative -- \"Use this skill for\" rather than \"this skill does\"\n- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.\n- The description competes with other skills for Claude's attention \u2014 make it distinctive and immediately recognizable.\n- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.\n\nI'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end. \n\nPlease respond with only the new description text in tags, nothing else.", - "response": "\nUse when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.\n\n", - "parsed_description": "Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.", - "char_count": 899, - "over_limit": false, - "final_description": "Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill." -} \ No newline at end of file diff --git a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_2.json b/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_2.json deleted file mode 100644 index f1a6e67..0000000 --- a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_2.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "iteration": 2, - "prompt": "You are optimizing a skill description for a Claude Code skill called \"reply-pr\". A \"skill\" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.\n\nThe description appears in Claude's \"available_skills\" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.\n\nHere's the current description:\n\n\"Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.\"\n\n\nCurrent scores (Train: 6/12):\n\nFAILED TO TRIGGER (should have triggered but didn't):\n - \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3 times)\n - \"triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?\" (triggered 0/3 times)\n - \"the claude bot reviewed my PR and left notes, help me draft replies and ping it back\" (triggered 0/3 times)\n - \"i already pushed the fixes for the review comments, now help me post replies referencing the commits\" (triggered 0/3 times)\n - \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly\" (triggered 0/3 times)\n - \"reply to the PR comments\" (triggered 0/3 times)\n\nPREVIOUS ATTEMPTS (do NOT repeat these \u2014 try something structurally different):\n\n\nDescription: \"Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.\"\nTrain results:\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\nDescription: \"Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.\"\nTrain results:\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\n\nSkill content (for context on what the skill does):\n\n---\nname: reply-pr\ndescription: >-\n Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots\n (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human\n deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the\n user wants to respond to, address, answer, or \"reply to\" PR review feedback;\n handle Codex/Claude/CodeRabbit comments; go through review comments one by one;\n or clear/resolve unresolved review threads \u2014 even if they don't name this skill\n and even if they just say \"let's reply to the PR comments\" or \"check the review\n comments on PR 1016\". Do NOT use it to actually fix the code the reviewers\n flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts\n the replies afterwards.\n---\n\n# reply-pr\n\nReviewers (bots and humans) leave comments on a PR. Answering them *well* means:\nreply on the **right channel for each reviewer**, go **item by item**, give a\n**rationale for both what you tackled and what you skipped**, and leave **no\nunresolved thread behind**. Doing that by hand every time is tedious and easy to\nget subtly wrong (wrong channel, missing ping, silent skips). This skill makes\nthe correct etiquette the default.\n\n## Two hard rules\n\n1. **Reply-only. Never edit code.** Any code fixes were already made in the\n normal dev flow; the replies just reference those commits. If the user hasn't\n made a fix yet, note it as a \"to fix\" in the gate \u2014 don't fix it here.\n2. **Never post without the deliberation gate.** Always stop, show the user your\n tackle/skip proposal, and wait for their steer before posting anything. This\n is the single most important behaviour \u2014 the user has repeatedly said \"do not\n action, just critically check\" / \"do not act, let me know first\".\n\n## The workflow \u2014 three phases\n\n### Phase 1 \u2014 Analyse (read-only)\n\n1. **Find the PR and know your role.** Use the argument if given (`reply-pr 1016`),\n else the open PR on the current branch. Get `owner/repo` and the PR number.\n `gather.sh` returns `self` (the login you post as) and `pr_author` (the PR's\n author). The **common case is `self == pr_author`** \u2014 you're the author\n answering reviewers; treat your own (`self`) comments as prior replies and skip\n them. But **don't assume it**: if `self != pr_author`, you're running as someone\n else (a reviewer or collaborator), so your own comments may be review *findings*,\n not replies \u2014 say so at the gate (\"running as ``, who isn't the author\n ``\") and confirm intent (replying on the author's behalf vs.\n responding to threads as a reviewer), then adapt. The user can also state their\n role in the prompt (e.g. \"I'm reviewing this, not the author\").\n2. **Gather the unresolved feedback.** Run the bundled script:\n `bash scripts/gather.sh [owner/repo]`\n It **fails loudly** on any fetch error (never a false \"no comments\"), paginates,\n and returns JSON with:\n - `self` \u2014 the login you post as (so you can recognise your own past replies) \u2014\n and `pr_author` \u2014 the PR's author (compare the two to know your role; step 1).\n - `unresolved_threads` \u2014 inline review threads with `isResolved == false`\n (filtered client-side; GraphQL has no server-side `isResolved` argument).\n Each has `thread_id` (for resolving), `comment_id` (the **root** comment, for\n `in_reply_to`), `reviewer` (bare login) + `is_bot`, `path`, `line`, `url`,\n `body` (the finding),\n `is_outdated`, **`on_changed_line`** (did this PR add/remove the flagged line,\n or is it unchanged context \u2014 a pre-existing hint; see step 4), and the\n thread's recent history: `comment_count`, `last_comment_author`,\n `last_comment_body`.\n - `top_level_comments` \u2014 PR conversation comments (where Claude-bot posts its\n whole review).\n Read `scripts/gather.sh` to see or adjust the queries.\n3. **Classify + dedup.** For each item, bucket it by reviewer and channel:\n - **Reviewer \u2192 bucket by `is_bot` + login.** `is_bot: false` is a **human**\n (highest priority). For bots (`is_bot: true`), match the login (already\n normalised \u2014 the `[bot]` suffix is stripped): `chatgpt-codex-connector` =\n Codex, `claude` = Claude, `coderabbitai` = CodeRabbit \u2014 use the matrix below.\n An **unknown bot** \u2192 default to an inline reply with **no ping** and flag it\n in the gate for the user to confirm the channel; never silently treat a bot\n as a human. (Classify by `is_bot`, not the login suffix: GitHub's GraphQL\n drops the `[bot]` suffix while REST keeps it, so the suffix is unreliable.)\n - **Dedup/group across reviewers** \u2014 multiple bots often flag the same thing.\n Present it once, noting who raised it.\n - **Drop CodeRabbit *nitpicks*.** Ignore anything CodeRabbit itself badges as a\n nitpick / trivial / minor. But a **substantive bug** CodeRabbit flags is NOT\n a nitpick \u2014 keep it.\n - **Skip what's already handled** (this is the only thing stopping a re-reply\n loop, since top-level comments have no \"resolved\" state). A thread whose\n `last_comment_author` is `self` is one we've already answered \u2014 don't\n re-reply. Among `top_level_comments`, ignore any authored by `self` (our own\n past replies), and skip a bot's review that a later comment already answers.\n4. **Critically assess each \u2014 don't take it at face value.** Verify the claim\n against the actual code/diff. Reviewers (especially bots) are often wrong,\n out of date, or flagging pre-existing code. Decide a **proposed stance**:\n **tackle** or **skip**, with a one-line rationale.\n - **Spotting pre-existing code (two signals \u2014 both advisory; you confirm at the\n gate, they never auto-skip):**\n - `on_changed_line: false` means the bot commented on an **unchanged context\n line** \u2014 code this PR did not touch. Strong \"pre-existing\" hint: lean toward\n skip-as-pre-existing (reply it's not part of this PR's work). `true`/`null`\n \u2192 treat as this PR's code and engage normally.\n - A finding can be a **source-PR duplicate** even on an added line \u2014 the PR\n re-includes code already reviewed in another (merged) PR, so it shows as\n *added* here and `on_changed_line` won't catch it. If you suspect it,\n confirm cheaply: find the commit behind the line (`git blame`, or the\n comment's commit) and run `gh api repos///commits//pulls`\n \u2014 if it returns another PR, the code came from there. Reply that it's from\n PR #X (link the originating thread) and skip.\n - Common skip rationales: *pre-existing / unrelated to this diff*, *out of scope\n / scope creep*, *accepted trade-off (flagged in the PR description for\n sign-off)*, or **already-shipped** (either case above \u2014 reply pointing at the\n origin, don't change anything here).\n\n### Phase 2 \u2014 Deliberation gate (STOP here)\n\nPresent a single ranked list for the user to steer. Order it **humans first**\n(their feedback outranks the bots), then Codex/Claude, then CodeRabbit. For each\nitem show: reviewer \u00b7 channel \u00b7 a one-line summary \u00b7 your proposed **tackle/skip**\n\u00b7 the rationale. Tag anything that looks **pre-existing** (`on_changed_line:\nfalse`, or a suspected source-PR duplicate) with a \"likely pre-existing\" marker so\nit stands out at a glance. Then **stop and wait.** Do not post anything yet.\n\nPhrase it as a proposal they edit, e.g. \"here's each unresolved comment with what\nI'd tackle vs skip and why \u2014 tell me what to change.\" If the user gave a mode\n(`reply-pr analyse`), stop here permanently and don't offer to post.\n\n### Phase 3 \u2014 Reply (only after the user approves)\n\nFor each item the user confirmed, post on the correct channel (matrix below),\n**item by item**, with the rationale \u2014 for **both** tackled and skipped items. A\nsilent skip is never acceptable; if we're not doing something, say why. For a\ntackled item, name the **commit SHA** and what changed.\n\n**Write each reply to a file first, then pass the file** to the helper \u2014\n`post-reply.sh` reads the body from a file **path** (that's what keeps backticks,\nquotes and newlines intact); passing the reply text directly will fail with \"No\nsuch file\". For example: write the reply to `/tmp/reply-.md`, then\n`bash scripts/post-reply.sh inline /tmp/reply-.md` (or\n`toplevel /tmp/reply-claude.md`). See the script header for all args.\n\nThen **resolve threads** (see \"Leaving no thread behind\" below).\n\n## How to reply to each reviewer\n\n**The general rule works for ANY reviewer** \u2014 known bot, unknown bot, or human.\nDon't hardcode to a fixed list; reply *in kind* and ping only when it helps:\n\n1. **Channel = reply where they commented.** An inline review comment \u2192 a threaded\n **inline** reply. A top-level PR comment \u2192 a **top-level** comment. `gather.sh`\n already splits these (`unresolved_threads` = inline, `top_level_comments` =\n top-level), so the right channel falls out for every reviewer automatically.\n2. **Ping = only reviewers known to want re-notifying** (default: **no ping**).\n Pinging re-triggers a bot to re-review \u2014 some want it, some don't, so when in\n doubt, don't.\n3. **Humans first** \u2014 a human's feedback outranks the bots; address it first.\n\n**Known-reviewer specifics.** Anything not listed falls back to the general rule\n(reply in kind, no ping) \u2014 and flag it in the gate so the user can confirm the\nchannel/ping for a bot you haven't seen before:\n\n| Reviewer (`is_bot` + login) | Channel | Ping |\n|---|---|---|\n| **Human** (`is_bot: false`) | in kind (inline or top-level) \u2014 **highest priority** | optional @-mention |\n| **Codex** (`chatgpt-codex-connector`) | inline, threaded under the finding | **`@codex`** |\n| **Claude** (`claude`) | **top-level** comment (it never comments inline) | **`@claude`** \u2014 reference the `#issuecomment-` URL, then go point by point |\n| **CodeRabbit** (`coderabbitai`) | inline (substantive items only; nitpicks dropped) | none |\n| **Any other bot** | in kind (wherever it commented) | none \u2014 flag in the gate to confirm |\n\n**Why Claude is the exception:** it posts its whole review as one top-level comment\nwith no inline threads, so a top-level `@claude` reply referencing the comment is\nthe only thing that lands. Every other reviewer (bots and humans) attaches findings\nto code lines, so threaded inline replies keep the conversation where they put it \u2014\nwhich is exactly what \"reply where they commented\" gives you for free.\n\n## Reply voice\n\nMatch the user's style (from replies he's posted verbatim). Substantive,\ntechnical, concise-but-complete. **No emojis.**\n\n- **Bold per-item headers with the verdict IN the header**, e.g.\n `**1. Nuxt tier-1 observation \u2014 skipped (intentionally, with rationale).**`\n- **Openers:** inline \u2192 `**Addressed in \\`\\`.**` ; top-level (Claude) \u2192\n `**@claude \u2014 this is a reply to your review at .** Going through your points one by one, with what we did and why.`\n- **Verdict vocabulary:** \"acknowledged, no code change\" \u00b7 \"agreed, no code\n change\" \u00b7 \"addressed\" \u00b7 \"correct observation, and we deliberately left it as-is\".\n- **Skip rationale leans on:** \"pre-existing \u2026 unrelated to this diff\" \u00b7 \"folding\n it into this PR would be scope creep\" \u00b7 \"out of scope\" \u00b7 \"flagged in the PR\n description for conscious sign-off\".\n- **Tackled:** name the SHA and what changed (\"Added a spec for a\n `.deployignore`-sourced culprit\").\n- **Closers:** a brief thank-you \u2014 \"Thanks for the catch.\"\n\n## Leaving no thread behind\n\nThe end state the user wants is **zero unresolved/pending threads**. Use each\nthread's `last_comment_author` / `last_comment_body` (from `gather.sh`) to decide\nwithout re-fetching:\n\n- After replying to an inline thread where **our reply is the last word** (a skip\n we've explained, or a fix we've described and nothing more is expected), resolve\n it: `bash scripts/post-reply.sh resolve `.\n- If `last_comment_author` is the **reviewer** (they replied after us \u2014 common\n with Codex, which often confirms), read `last_comment_body`: if it's a\n confirmation, resolve; if they raised a new point, handle it. If the thread is\n still **awaiting the reviewer's follow-up**, **do not resolve yet** \u2014 note it as\n \"waiting on \" and settle it on a later run. When in doubt, leave it\n open and flag it rather than resolving prematurely.\n- If a thread is `is_outdated: true`, the code moved since the comment was made,\n so the point may no longer apply \u2014 check whether it's still valid, say so in the\n reply if it's moot, and it's usually safe to resolve.\n\n## Scripts\n\n- `scripts/gather.sh [owner/repo]` \u2014 fetch unresolved review threads\n (paginated GraphQL, filtered client-side) + top-level comments as classified\n JSON, including `self` and each thread's recent history. Fails loudly on any\n fetch error rather than returning an empty result. Read it if you need to tweak\n the queries or bot-author filters.\n- `scripts/post-reply.sh ...` \u2014 post an inline reply (`inline `),\n a top-level comment (`toplevel `), or resolve a thread\n (`resolve `). It builds the JSON safely with `jq` and uses `gh api`.\n\nBoth require an authenticated `gh` CLI. The scripts are the deterministic\nplumbing; the judgement (classify, assess, gate, voice) stays here in the skill.\n\n\n\nBased on the failures, write a new and improved description that is more likely to trigger correctly. When I say \"based on the failures\", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:\n\n1. Avoid overfitting\n2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.\n\nConcretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters \u2014 descriptions over that will be truncated, so stay comfortably under it.\n\nHere are some tips that we've found to work well in writing these descriptions:\n- The skill should be phrased in the imperative -- \"Use this skill for\" rather than \"this skill does\"\n- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.\n- The description competes with other skills for Claude's attention \u2014 make it distinctive and immediately recognizable.\n- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.\n\nI'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end. \n\nPlease respond with only the new description text in tags, nothing else.", - "response": "\nResponding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\n\n", - "parsed_description": "Responding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.", - "char_count": 919, - "over_limit": false, - "final_description": "Responding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill." -} \ No newline at end of file diff --git a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_3.json b/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_3.json deleted file mode 100644 index f0188f8..0000000 --- a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_3.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "iteration": 3, - "prompt": "You are optimizing a skill description for a Claude Code skill called \"reply-pr\". A \"skill\" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.\n\nThe description appears in Claude's \"available_skills\" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.\n\nHere's the current description:\n\n\"Responding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\"\n\n\nCurrent scores (Train: 6/12):\n\nFAILED TO TRIGGER (should have triggered but didn't):\n - \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3 times)\n - \"triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?\" (triggered 0/3 times)\n - \"the claude bot reviewed my PR and left notes, help me draft replies and ping it back\" (triggered 0/3 times)\n - \"i already pushed the fixes for the review comments, now help me post replies referencing the commits\" (triggered 0/3 times)\n - \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly\" (triggered 0/3 times)\n - \"reply to the PR comments\" (triggered 0/3 times)\n\nPREVIOUS ATTEMPTS (do NOT repeat these \u2014 try something structurally different):\n\n\nDescription: \"Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.\"\nTrain results:\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\nDescription: \"Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.\"\nTrain results:\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\nDescription: \"Responding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\"\nTrain results:\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\n\nSkill content (for context on what the skill does):\n\n---\nname: reply-pr\ndescription: >-\n Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots\n (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human\n deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the\n user wants to respond to, address, answer, or \"reply to\" PR review feedback;\n handle Codex/Claude/CodeRabbit comments; go through review comments one by one;\n or clear/resolve unresolved review threads \u2014 even if they don't name this skill\n and even if they just say \"let's reply to the PR comments\" or \"check the review\n comments on PR 1016\". Do NOT use it to actually fix the code the reviewers\n flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts\n the replies afterwards.\n---\n\n# reply-pr\n\nReviewers (bots and humans) leave comments on a PR. Answering them *well* means:\nreply on the **right channel for each reviewer**, go **item by item**, give a\n**rationale for both what you tackled and what you skipped**, and leave **no\nunresolved thread behind**. Doing that by hand every time is tedious and easy to\nget subtly wrong (wrong channel, missing ping, silent skips). This skill makes\nthe correct etiquette the default.\n\n## Two hard rules\n\n1. **Reply-only. Never edit code.** Any code fixes were already made in the\n normal dev flow; the replies just reference those commits. If the user hasn't\n made a fix yet, note it as a \"to fix\" in the gate \u2014 don't fix it here.\n2. **Never post without the deliberation gate.** Always stop, show the user your\n tackle/skip proposal, and wait for their steer before posting anything. This\n is the single most important behaviour \u2014 the user has repeatedly said \"do not\n action, just critically check\" / \"do not act, let me know first\".\n\n## The workflow \u2014 three phases\n\n### Phase 1 \u2014 Analyse (read-only)\n\n1. **Find the PR and know your role.** Use the argument if given (`reply-pr 1016`),\n else the open PR on the current branch. Get `owner/repo` and the PR number.\n `gather.sh` returns `self` (the login you post as) and `pr_author` (the PR's\n author). The **common case is `self == pr_author`** \u2014 you're the author\n answering reviewers; treat your own (`self`) comments as prior replies and skip\n them. But **don't assume it**: if `self != pr_author`, you're running as someone\n else (a reviewer or collaborator), so your own comments may be review *findings*,\n not replies \u2014 say so at the gate (\"running as ``, who isn't the author\n ``\") and confirm intent (replying on the author's behalf vs.\n responding to threads as a reviewer), then adapt. The user can also state their\n role in the prompt (e.g. \"I'm reviewing this, not the author\").\n2. **Gather the unresolved feedback.** Run the bundled script:\n `bash scripts/gather.sh [owner/repo]`\n It **fails loudly** on any fetch error (never a false \"no comments\"), paginates,\n and returns JSON with:\n - `self` \u2014 the login you post as (so you can recognise your own past replies) \u2014\n and `pr_author` \u2014 the PR's author (compare the two to know your role; step 1).\n - `unresolved_threads` \u2014 inline review threads with `isResolved == false`\n (filtered client-side; GraphQL has no server-side `isResolved` argument).\n Each has `thread_id` (for resolving), `comment_id` (the **root** comment, for\n `in_reply_to`), `reviewer` (bare login) + `is_bot`, `path`, `line`, `url`,\n `body` (the finding),\n `is_outdated`, **`on_changed_line`** (did this PR add/remove the flagged line,\n or is it unchanged context \u2014 a pre-existing hint; see step 4), and the\n thread's recent history: `comment_count`, `last_comment_author`,\n `last_comment_body`.\n - `top_level_comments` \u2014 PR conversation comments (where Claude-bot posts its\n whole review).\n Read `scripts/gather.sh` to see or adjust the queries.\n3. **Classify + dedup.** For each item, bucket it by reviewer and channel:\n - **Reviewer \u2192 bucket by `is_bot` + login.** `is_bot: false` is a **human**\n (highest priority). For bots (`is_bot: true`), match the login (already\n normalised \u2014 the `[bot]` suffix is stripped): `chatgpt-codex-connector` =\n Codex, `claude` = Claude, `coderabbitai` = CodeRabbit \u2014 use the matrix below.\n An **unknown bot** \u2192 default to an inline reply with **no ping** and flag it\n in the gate for the user to confirm the channel; never silently treat a bot\n as a human. (Classify by `is_bot`, not the login suffix: GitHub's GraphQL\n drops the `[bot]` suffix while REST keeps it, so the suffix is unreliable.)\n - **Dedup/group across reviewers** \u2014 multiple bots often flag the same thing.\n Present it once, noting who raised it.\n - **Drop CodeRabbit *nitpicks*.** Ignore anything CodeRabbit itself badges as a\n nitpick / trivial / minor. But a **substantive bug** CodeRabbit flags is NOT\n a nitpick \u2014 keep it.\n - **Skip what's already handled** (this is the only thing stopping a re-reply\n loop, since top-level comments have no \"resolved\" state). A thread whose\n `last_comment_author` is `self` is one we've already answered \u2014 don't\n re-reply. Among `top_level_comments`, ignore any authored by `self` (our own\n past replies), and skip a bot's review that a later comment already answers.\n4. **Critically assess each \u2014 don't take it at face value.** Verify the claim\n against the actual code/diff. Reviewers (especially bots) are often wrong,\n out of date, or flagging pre-existing code. Decide a **proposed stance**:\n **tackle** or **skip**, with a one-line rationale.\n - **Spotting pre-existing code (two signals \u2014 both advisory; you confirm at the\n gate, they never auto-skip):**\n - `on_changed_line: false` means the bot commented on an **unchanged context\n line** \u2014 code this PR did not touch. Strong \"pre-existing\" hint: lean toward\n skip-as-pre-existing (reply it's not part of this PR's work). `true`/`null`\n \u2192 treat as this PR's code and engage normally.\n - A finding can be a **source-PR duplicate** even on an added line \u2014 the PR\n re-includes code already reviewed in another (merged) PR, so it shows as\n *added* here and `on_changed_line` won't catch it. If you suspect it,\n confirm cheaply: find the commit behind the line (`git blame`, or the\n comment's commit) and run `gh api repos///commits//pulls`\n \u2014 if it returns another PR, the code came from there. Reply that it's from\n PR #X (link the originating thread) and skip.\n - Common skip rationales: *pre-existing / unrelated to this diff*, *out of scope\n / scope creep*, *accepted trade-off (flagged in the PR description for\n sign-off)*, or **already-shipped** (either case above \u2014 reply pointing at the\n origin, don't change anything here).\n\n### Phase 2 \u2014 Deliberation gate (STOP here)\n\nPresent a single ranked list for the user to steer. Order it **humans first**\n(their feedback outranks the bots), then Codex/Claude, then CodeRabbit. For each\nitem show: reviewer \u00b7 channel \u00b7 a one-line summary \u00b7 your proposed **tackle/skip**\n\u00b7 the rationale. Tag anything that looks **pre-existing** (`on_changed_line:\nfalse`, or a suspected source-PR duplicate) with a \"likely pre-existing\" marker so\nit stands out at a glance. Then **stop and wait.** Do not post anything yet.\n\nPhrase it as a proposal they edit, e.g. \"here's each unresolved comment with what\nI'd tackle vs skip and why \u2014 tell me what to change.\" If the user gave a mode\n(`reply-pr analyse`), stop here permanently and don't offer to post.\n\n### Phase 3 \u2014 Reply (only after the user approves)\n\nFor each item the user confirmed, post on the correct channel (matrix below),\n**item by item**, with the rationale \u2014 for **both** tackled and skipped items. A\nsilent skip is never acceptable; if we're not doing something, say why. For a\ntackled item, name the **commit SHA** and what changed.\n\n**Write each reply to a file first, then pass the file** to the helper \u2014\n`post-reply.sh` reads the body from a file **path** (that's what keeps backticks,\nquotes and newlines intact); passing the reply text directly will fail with \"No\nsuch file\". For example: write the reply to `/tmp/reply-.md`, then\n`bash scripts/post-reply.sh inline /tmp/reply-.md` (or\n`toplevel /tmp/reply-claude.md`). See the script header for all args.\n\nThen **resolve threads** (see \"Leaving no thread behind\" below).\n\n## How to reply to each reviewer\n\n**The general rule works for ANY reviewer** \u2014 known bot, unknown bot, or human.\nDon't hardcode to a fixed list; reply *in kind* and ping only when it helps:\n\n1. **Channel = reply where they commented.** An inline review comment \u2192 a threaded\n **inline** reply. A top-level PR comment \u2192 a **top-level** comment. `gather.sh`\n already splits these (`unresolved_threads` = inline, `top_level_comments` =\n top-level), so the right channel falls out for every reviewer automatically.\n2. **Ping = only reviewers known to want re-notifying** (default: **no ping**).\n Pinging re-triggers a bot to re-review \u2014 some want it, some don't, so when in\n doubt, don't.\n3. **Humans first** \u2014 a human's feedback outranks the bots; address it first.\n\n**Known-reviewer specifics.** Anything not listed falls back to the general rule\n(reply in kind, no ping) \u2014 and flag it in the gate so the user can confirm the\nchannel/ping for a bot you haven't seen before:\n\n| Reviewer (`is_bot` + login) | Channel | Ping |\n|---|---|---|\n| **Human** (`is_bot: false`) | in kind (inline or top-level) \u2014 **highest priority** | optional @-mention |\n| **Codex** (`chatgpt-codex-connector`) | inline, threaded under the finding | **`@codex`** |\n| **Claude** (`claude`) | **top-level** comment (it never comments inline) | **`@claude`** \u2014 reference the `#issuecomment-` URL, then go point by point |\n| **CodeRabbit** (`coderabbitai`) | inline (substantive items only; nitpicks dropped) | none |\n| **Any other bot** | in kind (wherever it commented) | none \u2014 flag in the gate to confirm |\n\n**Why Claude is the exception:** it posts its whole review as one top-level comment\nwith no inline threads, so a top-level `@claude` reply referencing the comment is\nthe only thing that lands. Every other reviewer (bots and humans) attaches findings\nto code lines, so threaded inline replies keep the conversation where they put it \u2014\nwhich is exactly what \"reply where they commented\" gives you for free.\n\n## Reply voice\n\nMatch the user's style (from replies he's posted verbatim). Substantive,\ntechnical, concise-but-complete. **No emojis.**\n\n- **Bold per-item headers with the verdict IN the header**, e.g.\n `**1. Nuxt tier-1 observation \u2014 skipped (intentionally, with rationale).**`\n- **Openers:** inline \u2192 `**Addressed in \\`\\`.**` ; top-level (Claude) \u2192\n `**@claude \u2014 this is a reply to your review at .** Going through your points one by one, with what we did and why.`\n- **Verdict vocabulary:** \"acknowledged, no code change\" \u00b7 \"agreed, no code\n change\" \u00b7 \"addressed\" \u00b7 \"correct observation, and we deliberately left it as-is\".\n- **Skip rationale leans on:** \"pre-existing \u2026 unrelated to this diff\" \u00b7 \"folding\n it into this PR would be scope creep\" \u00b7 \"out of scope\" \u00b7 \"flagged in the PR\n description for conscious sign-off\".\n- **Tackled:** name the SHA and what changed (\"Added a spec for a\n `.deployignore`-sourced culprit\").\n- **Closers:** a brief thank-you \u2014 \"Thanks for the catch.\"\n\n## Leaving no thread behind\n\nThe end state the user wants is **zero unresolved/pending threads**. Use each\nthread's `last_comment_author` / `last_comment_body` (from `gather.sh`) to decide\nwithout re-fetching:\n\n- After replying to an inline thread where **our reply is the last word** (a skip\n we've explained, or a fix we've described and nothing more is expected), resolve\n it: `bash scripts/post-reply.sh resolve `.\n- If `last_comment_author` is the **reviewer** (they replied after us \u2014 common\n with Codex, which often confirms), read `last_comment_body`: if it's a\n confirmation, resolve; if they raised a new point, handle it. If the thread is\n still **awaiting the reviewer's follow-up**, **do not resolve yet** \u2014 note it as\n \"waiting on \" and settle it on a later run. When in doubt, leave it\n open and flag it rather than resolving prematurely.\n- If a thread is `is_outdated: true`, the code moved since the comment was made,\n so the point may no longer apply \u2014 check whether it's still valid, say so in the\n reply if it's moot, and it's usually safe to resolve.\n\n## Scripts\n\n- `scripts/gather.sh [owner/repo]` \u2014 fetch unresolved review threads\n (paginated GraphQL, filtered client-side) + top-level comments as classified\n JSON, including `self` and each thread's recent history. Fails loudly on any\n fetch error rather than returning an empty result. Read it if you need to tweak\n the queries or bot-author filters.\n- `scripts/post-reply.sh ...` \u2014 post an inline reply (`inline `),\n a top-level comment (`toplevel `), or resolve a thread\n (`resolve `). It builds the JSON safely with `jq` and uses `gh api`.\n\nBoth require an authenticated `gh` CLI. The scripts are the deterministic\nplumbing; the judgement (classify, assess, gate, voice) stays here in the skill.\n\n\n\nBased on the failures, write a new and improved description that is more likely to trigger correctly. When I say \"based on the failures\", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:\n\n1. Avoid overfitting\n2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.\n\nConcretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters \u2014 descriptions over that will be truncated, so stay comfortably under it.\n\nHere are some tips that we've found to work well in writing these descriptions:\n- The skill should be phrased in the imperative -- \"Use this skill for\" rather than \"this skill does\"\n- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.\n- The description competes with other skills for Claude's attention \u2014 make it distinctive and immediately recognizable.\n- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.\n\nI'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end. \n\nPlease respond with only the new description text in tags, nothing else.", - "response": "\nDraft and post replies to the review comments on a pull request, then resolve the threads. Use this whenever someone wants to answer, respond to, reply to, or triage the feedback reviewers left on a PR \u2014 whether from human reviewers or review bots like Codex, Claude, or CodeRabbit \u2014 instead of changing the code themselves. It decides which comments to address versus skip, writes and posts a reply for each (often citing the commits that already fixed things), pings bots back for re-review, and clears every unresolved thread so nothing is left hanging. Fires on asks like \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"help me draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\n\n", - "parsed_description": "Draft and post replies to the review comments on a pull request, then resolve the threads. Use this whenever someone wants to answer, respond to, reply to, or triage the feedback reviewers left on a PR \u2014 whether from human reviewers or review bots like Codex, Claude, or CodeRabbit \u2014 instead of changing the code themselves. It decides which comments to address versus skip, writes and posts a reply for each (often citing the commits that already fixed things), pings bots back for re-review, and clears every unresolved thread so nothing is left hanging. Fires on asks like \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"help me draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.", - "char_count": 1019, - "over_limit": false, - "final_description": "Draft and post replies to the review comments on a pull request, then resolve the threads. Use this whenever someone wants to answer, respond to, reply to, or triage the feedback reviewers left on a PR \u2014 whether from human reviewers or review bots like Codex, Claude, or CodeRabbit \u2014 instead of changing the code themselves. It decides which comments to address versus skip, writes and posts a reply for each (often citing the commits that already fixed things), pings bots back for re-review, and clears every unresolved thread so nothing is left hanging. Fires on asks like \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"help me draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill." -} \ No newline at end of file diff --git a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_4.json b/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_4.json deleted file mode 100644 index e138918..0000000 --- a/reply-pr-workspace/desc-opt/2026-07-06_151152/logs/improve_iter_4.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "iteration": 4, - "prompt": "You are optimizing a skill description for a Claude Code skill called \"reply-pr\". A \"skill\" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.\n\nThe description appears in Claude's \"available_skills\" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.\n\nHere's the current description:\n\n\"Draft and post replies to the review comments on a pull request, then resolve the threads. Use this whenever someone wants to answer, respond to, reply to, or triage the feedback reviewers left on a PR \u2014 whether from human reviewers or review bots like Codex, Claude, or CodeRabbit \u2014 instead of changing the code themselves. It decides which comments to address versus skip, writes and posts a reply for each (often citing the commits that already fixed things), pings bots back for re-review, and clears every unresolved thread so nothing is left hanging. Fires on asks like \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"help me draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\"\n\n\nCurrent scores (Train: 6/12):\n\nFAILED TO TRIGGER (should have triggered but didn't):\n - \"i already pushed the fixes for the review comments, now help me post replies referencing the commits\" (triggered 0/3 times)\n - \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3 times)\n - \"triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?\" (triggered 0/3 times)\n - \"the claude bot reviewed my PR and left notes, help me draft replies and ping it back\" (triggered 0/3 times)\n - \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly\" (triggered 0/3 times)\n - \"reply to the PR comments\" (triggered 0/3 times)\n\nPREVIOUS ATTEMPTS (do NOT repeat these \u2014 try something structurally different):\n\n\nDescription: \"Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.\"\nTrain results:\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\nDescription: \"Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.\"\nTrain results:\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\nDescription: \"Responding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\"\nTrain results:\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\nDescription: \"Draft and post replies to the review comments on a pull request, then resolve the threads. Use this whenever someone wants to answer, respond to, reply to, or triage the feedback reviewers left on a PR \u2014 whether from human reviewers or review bots like Codex, Claude, or CodeRabbit \u2014 instead of changing the code themselves. It decides which comments to address versus skip, writes and posts a reply for each (often citing the commits that already fixed things), pings bots back for re-review, and clears every unresolved thread so nothing is left hanging. Fires on asks like \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"help me draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.\"\nTrain results:\n [FAIL] \"i already pushed the fixes for the review comments, now help me post replies ref\" (triggered 0/3)\n [FAIL] \"respond to the reviewers on pull request 1200, don't leave anything unresolved\" (triggered 0/3)\n [FAIL] \"triage the review comments on #915 \u2014 which should I tackle vs skip before I repl\" (triggered 0/3)\n [FAIL] \"the claude bot reviewed my PR and left notes, help me draft replies and ping it \" (triggered 0/3)\n [FAIL] \"coderabbit and codex dumped a pile of comments on my PR #1042, help me go throug\" (triggered 0/3)\n [FAIL] \"reply to the PR comments\" (triggered 0/3)\n [PASS] \"review this PR and tell me what's wrong with it\" (triggered 0/3)\n [PASS] \"comment on this Linear ticket with the root cause we found\" (triggered 0/3)\n [PASS] \"write a PR description for the changes on this branch\" (triggered 0/3)\n [PASS] \"fix the bugs codex flagged in my PR #1042\" (triggered 0/3)\n [PASS] \"address the failing CI checks on my pr before I re-request review\" (triggered 0/3)\n [PASS] \"open a PR for my current branch and request review from the team\" (triggered 0/3)\n\n\n\n\nSkill content (for context on what the skill does):\n\n---\nname: reply-pr\ndescription: >-\n Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots\n (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human\n deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the\n user wants to respond to, address, answer, or \"reply to\" PR review feedback;\n handle Codex/Claude/CodeRabbit comments; go through review comments one by one;\n or clear/resolve unresolved review threads \u2014 even if they don't name this skill\n and even if they just say \"let's reply to the PR comments\" or \"check the review\n comments on PR 1016\". Do NOT use it to actually fix the code the reviewers\n flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts\n the replies afterwards.\n---\n\n# reply-pr\n\nReviewers (bots and humans) leave comments on a PR. Answering them *well* means:\nreply on the **right channel for each reviewer**, go **item by item**, give a\n**rationale for both what you tackled and what you skipped**, and leave **no\nunresolved thread behind**. Doing that by hand every time is tedious and easy to\nget subtly wrong (wrong channel, missing ping, silent skips). This skill makes\nthe correct etiquette the default.\n\n## Two hard rules\n\n1. **Reply-only. Never edit code.** Any code fixes were already made in the\n normal dev flow; the replies just reference those commits. If the user hasn't\n made a fix yet, note it as a \"to fix\" in the gate \u2014 don't fix it here.\n2. **Never post without the deliberation gate.** Always stop, show the user your\n tackle/skip proposal, and wait for their steer before posting anything. This\n is the single most important behaviour \u2014 the user has repeatedly said \"do not\n action, just critically check\" / \"do not act, let me know first\".\n\n## The workflow \u2014 three phases\n\n### Phase 1 \u2014 Analyse (read-only)\n\n1. **Find the PR and know your role.** Use the argument if given (`reply-pr 1016`),\n else the open PR on the current branch. Get `owner/repo` and the PR number.\n `gather.sh` returns `self` (the login you post as) and `pr_author` (the PR's\n author). The **common case is `self == pr_author`** \u2014 you're the author\n answering reviewers; treat your own (`self`) comments as prior replies and skip\n them. But **don't assume it**: if `self != pr_author`, you're running as someone\n else (a reviewer or collaborator), so your own comments may be review *findings*,\n not replies \u2014 say so at the gate (\"running as ``, who isn't the author\n ``\") and confirm intent (replying on the author's behalf vs.\n responding to threads as a reviewer), then adapt. The user can also state their\n role in the prompt (e.g. \"I'm reviewing this, not the author\").\n2. **Gather the unresolved feedback.** Run the bundled script:\n `bash scripts/gather.sh [owner/repo]`\n It **fails loudly** on any fetch error (never a false \"no comments\"), paginates,\n and returns JSON with:\n - `self` \u2014 the login you post as (so you can recognise your own past replies) \u2014\n and `pr_author` \u2014 the PR's author (compare the two to know your role; step 1).\n - `unresolved_threads` \u2014 inline review threads with `isResolved == false`\n (filtered client-side; GraphQL has no server-side `isResolved` argument).\n Each has `thread_id` (for resolving), `comment_id` (the **root** comment, for\n `in_reply_to`), `reviewer` (bare login) + `is_bot`, `path`, `line`, `url`,\n `body` (the finding),\n `is_outdated`, **`on_changed_line`** (did this PR add/remove the flagged line,\n or is it unchanged context \u2014 a pre-existing hint; see step 4), and the\n thread's recent history: `comment_count`, `last_comment_author`,\n `last_comment_body`.\n - `top_level_comments` \u2014 PR conversation comments (where Claude-bot posts its\n whole review).\n Read `scripts/gather.sh` to see or adjust the queries.\n3. **Classify + dedup.** For each item, bucket it by reviewer and channel:\n - **Reviewer \u2192 bucket by `is_bot` + login.** `is_bot: false` is a **human**\n (highest priority). For bots (`is_bot: true`), match the login (already\n normalised \u2014 the `[bot]` suffix is stripped): `chatgpt-codex-connector` =\n Codex, `claude` = Claude, `coderabbitai` = CodeRabbit \u2014 use the matrix below.\n An **unknown bot** \u2192 default to an inline reply with **no ping** and flag it\n in the gate for the user to confirm the channel; never silently treat a bot\n as a human. (Classify by `is_bot`, not the login suffix: GitHub's GraphQL\n drops the `[bot]` suffix while REST keeps it, so the suffix is unreliable.)\n - **Dedup/group across reviewers** \u2014 multiple bots often flag the same thing.\n Present it once, noting who raised it.\n - **Drop CodeRabbit *nitpicks*.** Ignore anything CodeRabbit itself badges as a\n nitpick / trivial / minor. But a **substantive bug** CodeRabbit flags is NOT\n a nitpick \u2014 keep it.\n - **Skip what's already handled** (this is the only thing stopping a re-reply\n loop, since top-level comments have no \"resolved\" state). A thread whose\n `last_comment_author` is `self` is one we've already answered \u2014 don't\n re-reply. Among `top_level_comments`, ignore any authored by `self` (our own\n past replies), and skip a bot's review that a later comment already answers.\n4. **Critically assess each \u2014 don't take it at face value.** Verify the claim\n against the actual code/diff. Reviewers (especially bots) are often wrong,\n out of date, or flagging pre-existing code. Decide a **proposed stance**:\n **tackle** or **skip**, with a one-line rationale.\n - **Spotting pre-existing code (two signals \u2014 both advisory; you confirm at the\n gate, they never auto-skip):**\n - `on_changed_line: false` means the bot commented on an **unchanged context\n line** \u2014 code this PR did not touch. Strong \"pre-existing\" hint: lean toward\n skip-as-pre-existing (reply it's not part of this PR's work). `true`/`null`\n \u2192 treat as this PR's code and engage normally.\n - A finding can be a **source-PR duplicate** even on an added line \u2014 the PR\n re-includes code already reviewed in another (merged) PR, so it shows as\n *added* here and `on_changed_line` won't catch it. If you suspect it,\n confirm cheaply: find the commit behind the line (`git blame`, or the\n comment's commit) and run `gh api repos///commits//pulls`\n \u2014 if it returns another PR, the code came from there. Reply that it's from\n PR #X (link the originating thread) and skip.\n - Common skip rationales: *pre-existing / unrelated to this diff*, *out of scope\n / scope creep*, *accepted trade-off (flagged in the PR description for\n sign-off)*, or **already-shipped** (either case above \u2014 reply pointing at the\n origin, don't change anything here).\n\n### Phase 2 \u2014 Deliberation gate (STOP here)\n\nPresent a single ranked list for the user to steer. Order it **humans first**\n(their feedback outranks the bots), then Codex/Claude, then CodeRabbit. For each\nitem show: reviewer \u00b7 channel \u00b7 a one-line summary \u00b7 your proposed **tackle/skip**\n\u00b7 the rationale. Tag anything that looks **pre-existing** (`on_changed_line:\nfalse`, or a suspected source-PR duplicate) with a \"likely pre-existing\" marker so\nit stands out at a glance. Then **stop and wait.** Do not post anything yet.\n\nPhrase it as a proposal they edit, e.g. \"here's each unresolved comment with what\nI'd tackle vs skip and why \u2014 tell me what to change.\" If the user gave a mode\n(`reply-pr analyse`), stop here permanently and don't offer to post.\n\n### Phase 3 \u2014 Reply (only after the user approves)\n\nFor each item the user confirmed, post on the correct channel (matrix below),\n**item by item**, with the rationale \u2014 for **both** tackled and skipped items. A\nsilent skip is never acceptable; if we're not doing something, say why. For a\ntackled item, name the **commit SHA** and what changed.\n\n**Write each reply to a file first, then pass the file** to the helper \u2014\n`post-reply.sh` reads the body from a file **path** (that's what keeps backticks,\nquotes and newlines intact); passing the reply text directly will fail with \"No\nsuch file\". For example: write the reply to `/tmp/reply-.md`, then\n`bash scripts/post-reply.sh inline /tmp/reply-.md` (or\n`toplevel /tmp/reply-claude.md`). See the script header for all args.\n\nThen **resolve threads** (see \"Leaving no thread behind\" below).\n\n## How to reply to each reviewer\n\n**The general rule works for ANY reviewer** \u2014 known bot, unknown bot, or human.\nDon't hardcode to a fixed list; reply *in kind* and ping only when it helps:\n\n1. **Channel = reply where they commented.** An inline review comment \u2192 a threaded\n **inline** reply. A top-level PR comment \u2192 a **top-level** comment. `gather.sh`\n already splits these (`unresolved_threads` = inline, `top_level_comments` =\n top-level), so the right channel falls out for every reviewer automatically.\n2. **Ping = only reviewers known to want re-notifying** (default: **no ping**).\n Pinging re-triggers a bot to re-review \u2014 some want it, some don't, so when in\n doubt, don't.\n3. **Humans first** \u2014 a human's feedback outranks the bots; address it first.\n\n**Known-reviewer specifics.** Anything not listed falls back to the general rule\n(reply in kind, no ping) \u2014 and flag it in the gate so the user can confirm the\nchannel/ping for a bot you haven't seen before:\n\n| Reviewer (`is_bot` + login) | Channel | Ping |\n|---|---|---|\n| **Human** (`is_bot: false`) | in kind (inline or top-level) \u2014 **highest priority** | optional @-mention |\n| **Codex** (`chatgpt-codex-connector`) | inline, threaded under the finding | **`@codex`** |\n| **Claude** (`claude`) | **top-level** comment (it never comments inline) | **`@claude`** \u2014 reference the `#issuecomment-` URL, then go point by point |\n| **CodeRabbit** (`coderabbitai`) | inline (substantive items only; nitpicks dropped) | none |\n| **Any other bot** | in kind (wherever it commented) | none \u2014 flag in the gate to confirm |\n\n**Why Claude is the exception:** it posts its whole review as one top-level comment\nwith no inline threads, so a top-level `@claude` reply referencing the comment is\nthe only thing that lands. Every other reviewer (bots and humans) attaches findings\nto code lines, so threaded inline replies keep the conversation where they put it \u2014\nwhich is exactly what \"reply where they commented\" gives you for free.\n\n## Reply voice\n\nMatch the user's style (from replies he's posted verbatim). Substantive,\ntechnical, concise-but-complete. **No emojis.**\n\n- **Bold per-item headers with the verdict IN the header**, e.g.\n `**1. Nuxt tier-1 observation \u2014 skipped (intentionally, with rationale).**`\n- **Openers:** inline \u2192 `**Addressed in \\`\\`.**` ; top-level (Claude) \u2192\n `**@claude \u2014 this is a reply to your review at .** Going through your points one by one, with what we did and why.`\n- **Verdict vocabulary:** \"acknowledged, no code change\" \u00b7 \"agreed, no code\n change\" \u00b7 \"addressed\" \u00b7 \"correct observation, and we deliberately left it as-is\".\n- **Skip rationale leans on:** \"pre-existing \u2026 unrelated to this diff\" \u00b7 \"folding\n it into this PR would be scope creep\" \u00b7 \"out of scope\" \u00b7 \"flagged in the PR\n description for conscious sign-off\".\n- **Tackled:** name the SHA and what changed (\"Added a spec for a\n `.deployignore`-sourced culprit\").\n- **Closers:** a brief thank-you \u2014 \"Thanks for the catch.\"\n\n## Leaving no thread behind\n\nThe end state the user wants is **zero unresolved/pending threads**. Use each\nthread's `last_comment_author` / `last_comment_body` (from `gather.sh`) to decide\nwithout re-fetching:\n\n- After replying to an inline thread where **our reply is the last word** (a skip\n we've explained, or a fix we've described and nothing more is expected), resolve\n it: `bash scripts/post-reply.sh resolve `.\n- If `last_comment_author` is the **reviewer** (they replied after us \u2014 common\n with Codex, which often confirms), read `last_comment_body`: if it's a\n confirmation, resolve; if they raised a new point, handle it. If the thread is\n still **awaiting the reviewer's follow-up**, **do not resolve yet** \u2014 note it as\n \"waiting on \" and settle it on a later run. When in doubt, leave it\n open and flag it rather than resolving prematurely.\n- If a thread is `is_outdated: true`, the code moved since the comment was made,\n so the point may no longer apply \u2014 check whether it's still valid, say so in the\n reply if it's moot, and it's usually safe to resolve.\n\n## Scripts\n\n- `scripts/gather.sh [owner/repo]` \u2014 fetch unresolved review threads\n (paginated GraphQL, filtered client-side) + top-level comments as classified\n JSON, including `self` and each thread's recent history. Fails loudly on any\n fetch error rather than returning an empty result. Read it if you need to tweak\n the queries or bot-author filters.\n- `scripts/post-reply.sh ...` \u2014 post an inline reply (`inline `),\n a top-level comment (`toplevel `), or resolve a thread\n (`resolve `). It builds the JSON safely with `jq` and uses `gh api`.\n\nBoth require an authenticated `gh` CLI. The scripts are the deterministic\nplumbing; the judgement (classify, assess, gate, voice) stays here in the skill.\n\n\n\nBased on the failures, write a new and improved description that is more likely to trigger correctly. When I say \"based on the failures\", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:\n\n1. Avoid overfitting\n2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.\n\nConcretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters \u2014 descriptions over that will be truncated, so stay comfortably under it.\n\nHere are some tips that we've found to work well in writing these descriptions:\n- The skill should be phrased in the imperative -- \"Use this skill for\" rather than \"this skill does\"\n- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.\n- The description competes with other skills for Claude's attention \u2014 make it distinctive and immediately recognizable.\n- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.\n\nI'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end. \n\nPlease respond with only the new description text in tags, nothing else.", - "response": "\nAnswer the reviewers on a pull request and clear the threads \u2014 reply, don't recode. Use whenever someone wants to respond to, reply to, answer, or triage the review comments left on a PR, whether by human reviewers or review bots like Codex, Claude, or CodeRabbit. It decides which comments to address vs skip, drafts and posts a reply on each thread (often citing the commits that already fixed things), pings bots back for re-review, and resolves every unresolved thread so nothing is left hanging. Triggers on \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply\", \"draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 fixing what a reviewer caught is a separate coding task.\n\n", - "parsed_description": "Answer the reviewers on a pull request and clear the threads \u2014 reply, don't recode. Use whenever someone wants to respond to, reply to, answer, or triage the review comments left on a PR, whether by human reviewers or review bots like Codex, Claude, or CodeRabbit. It decides which comments to address vs skip, drafts and posts a reply on each thread (often citing the commits that already fixed things), pings bots back for re-review, and resolves every unresolved thread so nothing is left hanging. Triggers on \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply\", \"draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 fixing what a reviewer caught is a separate coding task.", - "char_count": 879, - "over_limit": false, - "final_description": "Answer the reviewers on a pull request and clear the threads \u2014 reply, don't recode. Use whenever someone wants to respond to, reply to, answer, or triage the review comments left on a PR, whether by human reviewers or review bots like Codex, Claude, or CodeRabbit. It decides which comments to address vs skip, drafts and posts a reply on each thread (often citing the commits that already fixed things), pings bots back for re-review, and resolves every unresolved thread so nothing is left hanging. Triggers on \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply\", \"draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 fixing what a reviewer caught is a separate coding task." -} \ No newline at end of file diff --git a/reply-pr-workspace/desc-opt/2026-07-06_151152/results.json b/reply-pr-workspace/desc-opt/2026-07-06_151152/results.json deleted file mode 100644 index 4bb4460..0000000 --- a/reply-pr-workspace/desc-opt/2026-07-06_151152/results.json +++ /dev/null @@ -1,1390 +0,0 @@ -{ - "exit_reason": "max_iterations (5)", - "original_description": "Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.", - "best_description": "Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.", - "best_score": "4/8", - "best_train_score": "6/12", - "best_test_score": "4/8", - "final_description": "Answer the reviewers on a pull request and clear the threads \u2014 reply, don't recode. Use whenever someone wants to respond to, reply to, answer, or triage the review comments left on a PR, whether by human reviewers or review bots like Codex, Claude, or CodeRabbit. It decides which comments to address vs skip, drafts and posts a reply on each thread (often citing the commits that already fixed things), pings bots back for re-review, and resolves every unresolved thread so nothing is left hanging. Triggers on \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply\", \"draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 fixing what a reviewer caught is a separate coding task.", - "iterations_run": 5, - "holdout": 0.4, - "train_size": 12, - "test_size": 8, - "history": [ - { - "iteration": 1, - "description": "Triage and reply to a pull request's UNRESOLVED review comments \u2014 from bots (Codex, Claude, CodeRabbit) and human reviewers \u2014 with a mandatory human deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the user wants to respond to, address, answer, or \"reply to\" PR review feedback; handle Codex/Claude/CodeRabbit comments; go through review comments one by one; or clear/resolve unresolved review threads \u2014 even if they don't name this skill and even if they just say \"let's reply to the PR comments\" or \"check the review comments on PR 1016\". Do NOT use it to actually fix the code the reviewers flagged \u2014 fixes happen in the normal dev flow; this skill only crafts and posts the replies afterwards.", - "train_passed": 6, - "train_failed": 6, - "train_total": 12, - "train_results": [ - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "test_passed": 4, - "test_failed": 4, - "test_total": 8, - "test_results": [ - { - "query": "30-ish coderabbit comments on my PR, mostly nitpicks \u2014 help me answer the real ones and skip the noise", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "there are a bunch of unresolved threads on pr 883, i wanna answer them all and get to zero unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "let's respond to the review feedback on this pr \u2014 reply to each reviewer the right way", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "go through codex's review on the current branch's PR and write responses for me, humans first", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to my manager's slack thread about the staging deploy", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "resolve the merge conflicts on PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "leave a comment on PR #900 that it's good to merge", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "what did coderabbit flag on my PR? just summarize it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "passed": 6, - "failed": 6, - "total": 12, - "results": [ - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ] - }, - { - "iteration": 2, - "description": "Use when the user wants to respond to, answer, or reply to the review comments on a pull request \u2014 whether left by human reviewers or by bots like Codex, Claude, or CodeRabbit. Handles triaging which comments to address versus skip, drafting and posting the replies (often citing the commits that already fixed things), pinging bots back for re-review, and resolving unresolved review threads so nothing is left hanging. Triggers on requests like \"reply to the PR comments\", \"respond to the reviewers on PR 1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex comments and reply properly\", \"help me draft replies to the bot and ping it back\", or \"triage the review feedback before I reply\". REPLY-ONLY: it crafts and posts the replies but never edits the flagged code \u2014 actual code fixes happen separately in the normal dev flow, so \"fix the bug Codex flagged\" is NOT this skill.", - "train_passed": 6, - "train_failed": 6, - "train_total": 12, - "train_results": [ - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "test_passed": 4, - "test_failed": 4, - "test_total": 8, - "test_results": [ - { - "query": "30-ish coderabbit comments on my PR, mostly nitpicks \u2014 help me answer the real ones and skip the noise", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "there are a bunch of unresolved threads on pr 883, i wanna answer them all and get to zero unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "let's respond to the review feedback on this pr \u2014 reply to each reviewer the right way", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "go through codex's review on the current branch's PR and write responses for me, humans first", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to my manager's slack thread about the staging deploy", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "resolve the merge conflicts on PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "what did coderabbit flag on my PR? just summarize it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "leave a comment on PR #900 that it's good to merge", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "passed": 6, - "failed": 6, - "total": 12, - "results": [ - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ] - }, - { - "iteration": 3, - "description": "Responding to code-review feedback on a pull request. Use whenever the user wants to reply to, answer, address, or triage the review comments on a PR \u2014 left by human reviewers or by review bots like Codex, Claude, or CodeRabbit \u2014 rather than rewrite the code. Typical asks: \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"the claude bot reviewed my PR, help me draft replies and ping it back\", \"triage which comments to tackle vs skip before I reply\", or \"I already pushed the fixes, now post replies citing the commits\". It decides what to address versus skip, drafts and posts the replies, pings bots for re-review, and resolves leftover review threads. It does NOT edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.", - "train_passed": 6, - "train_failed": 6, - "train_total": 12, - "train_results": [ - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "test_passed": 4, - "test_failed": 4, - "test_total": 8, - "test_results": [ - { - "query": "there are a bunch of unresolved threads on pr 883, i wanna answer them all and get to zero unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "30-ish coderabbit comments on my PR, mostly nitpicks \u2014 help me answer the real ones and skip the noise", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "let's respond to the review feedback on this pr \u2014 reply to each reviewer the right way", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "go through codex's review on the current branch's PR and write responses for me, humans first", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to my manager's slack thread about the staging deploy", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "resolve the merge conflicts on PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "leave a comment on PR #900 that it's good to merge", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "what did coderabbit flag on my PR? just summarize it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "passed": 6, - "failed": 6, - "total": 12, - "results": [ - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ] - }, - { - "iteration": 4, - "description": "Draft and post replies to the review comments on a pull request, then resolve the threads. Use this whenever someone wants to answer, respond to, reply to, or triage the feedback reviewers left on a PR \u2014 whether from human reviewers or review bots like Codex, Claude, or CodeRabbit \u2014 instead of changing the code themselves. It decides which comments to address versus skip, writes and posts a reply for each (often citing the commits that already fixed things), pings bots back for re-review, and clears every unresolved thread so nothing is left hanging. Fires on asks like \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply properly\", \"help me draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 actually fixing what a reviewer caught (e.g. \"fix the bug Codex flagged\") is a separate coding task, not this skill.", - "train_passed": 6, - "train_failed": 6, - "train_total": 12, - "train_results": [ - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "test_passed": 4, - "test_failed": 4, - "test_total": 8, - "test_results": [ - { - "query": "30-ish coderabbit comments on my PR, mostly nitpicks \u2014 help me answer the real ones and skip the noise", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "there are a bunch of unresolved threads on pr 883, i wanna answer them all and get to zero unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "go through codex's review on the current branch's PR and write responses for me, humans first", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "let's respond to the review feedback on this pr \u2014 reply to each reviewer the right way", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to my manager's slack thread about the staging deploy", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "leave a comment on PR #900 that it's good to merge", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "resolve the merge conflicts on PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "what did coderabbit flag on my PR? just summarize it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "passed": 6, - "failed": 6, - "total": 12, - "results": [ - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ] - }, - { - "iteration": 5, - "description": "Answer the reviewers on a pull request and clear the threads \u2014 reply, don't recode. Use whenever someone wants to respond to, reply to, answer, or triage the review comments left on a PR, whether by human reviewers or review bots like Codex, Claude, or CodeRabbit. It decides which comments to address vs skip, drafts and posts a reply on each thread (often citing the commits that already fixed things), pings bots back for re-review, and resolves every unresolved thread so nothing is left hanging. Triggers on \"reply to the PR comments\", \"respond to the reviewers on #1200 and leave nothing unresolved\", \"go through the CodeRabbit and Codex notes and reply\", \"draft replies to the claude bot and ping it back\", or \"triage which review comments to tackle vs skip before I reply\". Do NOT use it to edit the flagged code \u2014 fixing what a reviewer caught is a separate coding task.", - "train_passed": 6, - "train_failed": 6, - "train_total": 12, - "train_results": [ - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "test_passed": 4, - "test_failed": 4, - "test_total": 8, - "test_results": [ - { - "query": "30-ish coderabbit comments on my PR, mostly nitpicks \u2014 help me answer the real ones and skip the noise", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "there are a bunch of unresolved threads on pr 883, i wanna answer them all and get to zero unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "let's respond to the review feedback on this pr \u2014 reply to each reviewer the right way", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "go through codex's review on the current branch's PR and write responses for me, humans first", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to my manager's slack thread about the staging deploy", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "what did coderabbit flag on my PR? just summarize it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "leave a comment on PR #900 that it's good to merge", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "resolve the merge conflicts on PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ], - "passed": 6, - "failed": 6, - "total": 12, - "results": [ - { - "query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "triage the review comments on #915 \u2014 which should I tackle vs skip before I reply?", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "reply to the PR comments", - "should_trigger": true, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": false - }, - { - "query": "review this PR and tell me what's wrong with it", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "comment on this Linear ticket with the root cause we found", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "write a PR description for the changes on this branch", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "fix the bugs codex flagged in my PR #1042", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "address the failing CI checks on my pr before I re-request review", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - }, - { - "query": "open a PR for my current branch and request review from the team", - "should_trigger": false, - "trigger_rate": 0.0, - "triggers": 0, - "runs": 3, - "pass": true - } - ] - } - ] -} \ No newline at end of file diff --git a/reply-pr-workspace/trigger-eval.json b/reply-pr-workspace/trigger-eval.json deleted file mode 100644 index 72f7a80..0000000 --- a/reply-pr-workspace/trigger-eval.json +++ /dev/null @@ -1,22 +0,0 @@ -[ - {"query": "coderabbit and codex dumped a pile of comments on my PR #1042, help me go through them and reply properly", "should_trigger": true}, - {"query": "reply to the PR comments", "should_trigger": true}, - {"query": "let's respond to the review feedback on this pr — reply to each reviewer the right way", "should_trigger": true}, - {"query": "there are a bunch of unresolved threads on pr 883, i wanna answer them all and get to zero unresolved", "should_trigger": true}, - {"query": "the claude bot reviewed my PR and left notes, help me draft replies and ping it back", "should_trigger": true}, - {"query": "i already pushed the fixes for the review comments, now help me post replies referencing the commits", "should_trigger": true}, - {"query": "triage the review comments on #915 — which should I tackle vs skip before I reply?", "should_trigger": true}, - {"query": "30-ish coderabbit comments on my PR, mostly nitpicks — help me answer the real ones and skip the noise", "should_trigger": true}, - {"query": "go through codex's review on the current branch's PR and write responses for me, humans first", "should_trigger": true}, - {"query": "respond to the reviewers on pull request 1200, don't leave anything unresolved", "should_trigger": true}, - {"query": "fix the bugs codex flagged in my PR #1042", "should_trigger": false}, - {"query": "review this PR and tell me what's wrong with it", "should_trigger": false}, - {"query": "leave a comment on PR #900 that it's good to merge", "should_trigger": false}, - {"query": "reply to my manager's slack thread about the staging deploy", "should_trigger": false}, - {"query": "what did coderabbit flag on my PR? just summarize it", "should_trigger": false}, - {"query": "resolve the merge conflicts on PR #1042", "should_trigger": false}, - {"query": "address the failing CI checks on my pr before I re-request review", "should_trigger": false}, - {"query": "write a PR description for the changes on this branch", "should_trigger": false}, - {"query": "comment on this Linear ticket with the root cause we found", "should_trigger": false}, - {"query": "open a PR for my current branch and request review from the team", "should_trigger": false} -] diff --git a/skills/reply-pr/SKILL.md b/skills/reply-pr/SKILL.md index 0bc7c78..9f30ded 100644 --- a/skills/reply-pr/SKILL.md +++ b/skills/reply-pr/SKILL.md @@ -1,16 +1,16 @@ --- name: reply-pr description: >- - Triage and reply to a pull request's UNRESOLVED review comments — from bots - (Codex, Claude, CodeRabbit) and human reviewers — with a mandatory human - deliberation gate. It is REPLY-ONLY: it never edits code. Use it whenever the - user wants to respond to, address, answer, or "reply to" PR review feedback; - handle Codex/Claude/CodeRabbit comments; go through review comments one by one; - or clear/resolve unresolved review threads — even if they don't name this skill - and even if they just say "let's reply to the PR comments" or "check the review - comments on PR 1016". Do NOT use it to actually fix the code the reviewers - flagged — fixes happen in the normal dev flow; this skill only crafts and posts - the replies afterwards. + Answer a PR's review comments the right way, in one pass — triage the UNRESOLVED + feedback from review bots (Codex, Claude, CodeRabbit, or any other) and human + reviewers, then post a reply on each reviewer's own channel in your voice and + resolve every thread so nothing is left hanging. Reply-only: it never edits code, + and never posts anything without your approval. Use it whenever someone wants to + respond to, reply to, answer, or triage the review comments on a PR — even phrased + loosely, like "reply to the PR comments", "go through the CodeRabbit and Codex + notes and reply", "respond to the reviewers on #1200 and leave nothing unresolved", + or "which review comments should I tackle vs skip?". Do NOT use it to edit the code + a reviewer flagged — fixing what a reviewer caught is a separate coding task. --- # reply-pr