CLI-507 Add e2e tests for sonar-context-augmentation#317
Draft
mpaladin wants to merge 3 commits into
Draft
Conversation
mpaladin
changed the title
Contributor
Author
|
@sonar-review-alpha review |
Seeds a stale-version CAG skill plus a stale config.cliVersion in state.json so runPostUpdateActions() re-downloads the real binary from binaries.sonarsource.com, verifies its pinned PGP signature, extracts the tarball, and reruns `tool install-skill` — covering the real-binary surface that does not need SonarQube/Cloud project access. Asserts on the on-disk binary, refreshed skill version, and `sonar context --help` forwarding.
Adds a SKILL.md assertion to the initial post-update path and a nested describe that simulates a second self-update by rewinding state versions and deleting the rendered skill. The rerun must recreate SKILL.md with identical content and re-bump both cliVersion and the recorded skill version, proving `tool install-skill` actually runs (not just a state write).
Adds three new offline e2e files plus a shared helpers module: - cag-edge-cases.test.ts: deleted projectRoot, global skill, multi-skill refresh in one pass, no-op when cliVersion is current, stale-version binary cleanup in <binDir>. - cag-passthrough.test.ts: unauthenticated action surfaces the CLI's "Not authenticated" error, bare `sonar context` falls through to CAG's help, child non-zero exit propagates to the parent. - cag-copilot.test.ts: post-update refreshes a recorded copilot-cli skill and writes .github/skills/sonar-context-augmentation/SKILL.md. The integrate guard rails (CAG skipped on SonarQube Server, skipped with --global) are intentionally left to the existing integration tests at tests/integration/specs/integrate/context-augmentation.test.ts since the real binary never runs in those paths — nothing new for e2e to cover.
|
mpaladin
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an offline e2e suite under
tests/e2e/context/that exercises the realsonar-context-augmentationbinary — download, PGP signature verification, tar extraction, skill rendering, post-update refresh, and passthrough — without touching SonarQube/Cloud. The only network reach isbinaries.sonarsource.comfor the archive and detached signature.The trigger path uses pre-seeded
state.json(staleconfig.cliVersion+ stale-version skill record) sorunPostUpdateActions()fires and walks the real install chain, bypassing the entitlement check andcag initthat would otherwise require a live SQ Cloud connection.Coverage
cag-offline.test.ts--version, state bump,SKILL.mdwritten,sonar context --helppassthrough) + simulated self-update reinstallcag-edge-cases.test.tscag-passthrough.test.tscag-copilot.test.ts.github/skills/sonar-context-augmentation/SKILL.mdfor acopilot-cliskill record_helpers.tsbuildSkillExtension/seedState/findRecordedCagSkillscaffoldingTotal: 21 tests in ~6.3s locally.
Not in scope
The two `integrate` guard rails I considered (CAG skipped on SonarQube Server, CAG skipped with `--global`) are already covered by integration tests at
tests/integration/specs/integrate/context-augmentation.test.ts:454,570. Since those paths return before the binary would be downloaded, an e2e variant exercises no real-binary code that integration doesn't already cover.Live SonarQube Cloud coverage (entitlement check, `cag init`, real `sonar context `) is deferred — those require Vault secrets and a CAG-entitled test org.