Skip to content

CI: post VSIX download + size-delta comment on PRs#53

Open
nmetulev wants to merge 1 commit into
mainfrom
nmetulev-ci-vsix-size-comment
Open

CI: post VSIX download + size-delta comment on PRs#53
nmetulev wants to merge 1 commit into
mainfrom
nmetulev-ci-vsix-size-comment

Conversation

@nmetulev

Copy link
Copy Markdown
Member

Description

Adds a PR bot comment that posts the built extension VSIX download link and its size delta vs. the main baseline — mirroring the pattern the sibling repo microsoft/winappcli already uses (post-metrics-comment.yml), scoped down to our single VSIX artifact (no CLI/MSIX/NuGet/test/coverage/startup metrics).

The Build and Test workflow already builds and uploads the extension as the vscode-extension artifact, but never surfaced its size on PRs. Now it does.

How it works

  1. build.yml (additive): after Package VSIX, a step measures the VSIX into artifacts/vsix-sizes.json ({"vsix": <bytes>}). On push to main it saves that as an immutable-cache baseline (winappvsce-vsix-baseline, deleting the old entry first since caches are immutable). On pull_request it restores the baseline and uploads a metrics-data artifact (current size + baseline + pr-context.json).
  2. post-vsix-comment.yml (new): triggered by workflow_run on Build and Test, so it runs in the base repo context and has pull-requests: write even for fork PRs — that split is the whole point. It downloads metrics-data, resolves the PR, and creates/updates one idempotent comment.
  3. mark-vsix-stale.yml (new): on pull_request_target, prepends a "build in progress" banner to the existing comment when a new commit lands.

Security model (preserved from winappcli)

The metrics-data artifact is produced by the PR's own code, so it is treated as untrusted. The comment workflow:

  • uses only trusted workflow_run metadata (head_sha, run id) — never the artifact's pr-context.json;
  • resolves the PR via listPullRequestsAssociatedWithCommit and verifies the PR head still matches (an older run won't clobber a newer commit's report);
  • allowlists the single vsix key and coerces values (safeNum), so a malicious build can't inject table breaks, mentions, hidden HTML, or links.

Usage Example

Rendered comment on a PR (marker <!-- vsix-size-report -->, updated in place):

VSIX Build

Artifact Baseline Current Delta
VS Code Extension (VSIX) 3.24 MB 3.40 MB 📈 +164.0 KB (+4.94%)

On the first-ever PR (no baseline yet) it falls back to an absolute-size-only table.

Related Issue

Type of Change

  • 🔧 Config/build

Checklist

  • package.json contribution points / command/debugger/custom-editor docs updated (if extension surface changed) — N/A, CI-only change
  • Tested locally on Windows (validated YAML + embedded JS/pwsh + a mocked end-to-end render; see Additional Notes)

Additional Notes

⚠️ Bootstrap caveat: workflow_run / pull_request_target comment workflows only run from the version on the default branch. So the first live comment appears on the next PR after this one merges — this PR itself will upload metrics-data but won't get a comment, because post-vsix-comment.yml isn't on main yet. Likewise the main baseline cache is first written when this merges to main, so the earliest PRs will show the absolute-size fallback until a baseline exists.

Permissions: the build job gains actions: write (needed for gh cache delete on main). Commenting deliberately stays out of the build job — it happens only in the separate privileged workflow. The vscode-extension artifact is reused as-is (not renamed).

Validation performed locally:

  • yaml.safe_load on all three workflow files;
  • node --check on both actions/github-script blocks;
  • PowerShell parser ([Parser]::ParseInput) on every new pwsh block;
  • a mocked end-to-end run of the comment script covering the baseline, no-baseline, and update paths, and confirming an injected unknown key is dropped by the allowlist.

AI Description

This section is auto-generated by AI when the PR is opened or updated. To opt out, delete this entire section including the marker comments.

Mirror microsoft/winappcli's post-metrics-comment pattern, scoped to the single
VSIX artifact (no CLI/MSIX/NuGet/test/coverage/startup metrics).

- build.yml (additive): measure the packaged VSIX into artifacts/vsix-sizes.json
  ({"vsix": <bytes>}); on push to main save it as an immutable-cache baseline
  (winappvsce-vsix-baseline, deleting the old entry first); on pull_request
  restore the baseline and upload a metrics-data artifact (current + baseline +
  pr-context). Adds actions:write for `gh cache delete`; keeps the existing
  vscode-extension artifact and every prior step unchanged.
- post-vsix-comment.yml (new): workflow_run on "Build and Test" runs in the base
  repo so it has pull-requests:write even for fork PRs. Preserves the winappcli
  security model - trusts only workflow_run head_sha + run id, resolves/verifies
  the PR via listPullRequestsAssociatedWithCommit, allowlists the single `vsix`
  key, coerces values with safeNum. Renders an idempotent "## VSIX Build" comment
  (Baseline | Current | Delta) linking the row to the vscode-extension artifact
  download URL, with an absolute-size fallback on the first PR.
- mark-vsix-stale.yml (new): pull_request_target banner marking the comment
  stale while a new build runs.

Validated: yaml.safe_load on all three files, node --check on both
github-script blocks, PowerShell parser on all new pwsh blocks, plus a mocked
end-to-end render of the comment (baseline / no-baseline / update paths, and
the allowlist dropping unknown keys).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b889e1ad-e208-4032-a52a-c3ed2429c58b
@nmetulev nmetulev marked this pull request as ready for review July 14, 2026 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant