feat(meta): add agent-review rubric and example GitHub Action#1209
feat(meta): add agent-review rubric and example GitHub Action#1209jfindlay wants to merge 1 commit into
Conversation
* `AGENTS-REVIEW.md`: A vendor-neutral, rubric for a single review pass on a Physlib PR. * `.github/workflows/agent-review.yml`: An example implementation triggered by the `agent-review` label. Co-authored-by: Claude Sonnet 4.6 <noreply+claude-sonnet@anthropic.com>
|
Thank you for this PR, which will now be reviewed. If you are submitting to ./PhyslibAlpha there will be a lighter review process, If you want to bring attention to this PR, please write a message on this |
| AGENT_REVIEW_API_KEY: ${{ secrets.AGENT_REVIEW_API_KEY }} | ||
| # Configure the model here. Default: a mid-tier model. | ||
| # Escalate to a top-tier model only for large or complex PRs by exception. | ||
| AGENT_MODEL: "claude-sonnet-4-5" |
There was a problem hiding this comment.
sonnet 4.5 is not good enough. I think we should use the top tier models here like opus-4.8 or gpt-5.5
There was a problem hiding this comment.
Do we know who actually provides the agents for this? I assume we would need some AI key.
This might be a use case of #1211
There was a problem hiding this comment.
sonnet 4.5 is not good enough. I think we should use the top tier models here like opus-4.8 or gpt-5.5
Sonnet 4.5 is a compromise on token cost. There are some other token optimizations in AGENTS-REVIEW.md as it's currently written, like only running the review on explicit (re)label. I agree that some aspects of the review would be better at a higher tier agent, but more mechanical tasks like assessing docs coverage could be given to a lower tier agent to save cost.
Do we know who actually provides the agents for this? I assume we would need some AI key.
Proprietary frontier agent models are less generous with their free tier than for example GitHub itself. I do not know much about this area, but since this is a high profile, important, public project, there could be a way to get a sponsorship or a discount or perhaps GitHub or some libre software consortium has a program for such.
There was a problem hiding this comment.
Considering the current PR traffic of PhysLib, I think there is no need to compromise on token cost. I also generally do not trust models weaker than Opus-4.6; they sometimes produce serious hallucinations and require a lot of engineering effort to make them work properly.
I am still reading the GitHub agentic workflows documentation. It seems like it supports AI model subscription tiers, in which case a 100-200usd/month subscription should definitely be enough.
AGENTS-REVIEW.md: A vendor-neutral, rubric for a single review pass on a Physlib PR..github/workflows/agent-review.yml: An example implementation triggered by theagent-reviewlabel.