Skip to content

SMOKE_TEST.md

Latest commit

 

History

History
151 lines (113 loc) · 5.93 KB

SMOKE_TEST.md

File metadata and controls

151 lines (113 loc) · 5.93 KB

Smoke Test

The smoke test (smoke-test/) is a standalone npm package that runs a live read sweep against a real Vanta API tenant on every Speakeasy regeneration PR. It catches the class of bug that build-green can't: spec drift, OAuth-endpoint regressions, wrong server URLs, and encoding regressions in new Speakeasy CLI versions — before they ship to customers as a broken npm release.

The unit/integration tests Speakeasy generates under src/ mock the HTTP layer; this subproject is genuinely additive and only exists in CI on PRs to main.

Fixture

Smoke tests run against a long-lived audit on the production Vanta tenant, auth'd via a demo audit-firm account managed in 1Password.
Customer Amalgamated Widgets (Vanta's standard demo customer)
Framework SOC 2 Type II
Audit ID 6a01f08107cb79d9ad9645cb
Audit window 2026-05-10 → 2040-05-10
API base https://api.vanta.com
OAuth client created via Developer Console on the demo audit-firm account

The fixture audit is intentionally sparse today — Information Requests, comments, evidence, snapshots, framework-codes are empty. Endpoints that need non-empty children (e.g. getEvidenceUrls after listEvidence) report as skipped rather than failed; see Empty fixture data.

Secrets

GitHub Actions secrets, set at the repo level on both vanta-auditor-api-sdk-typescript and vanta-auditor-api-sdk-java:

  • SMOKE_OAUTH_CLIENT_ID
  • SMOKE_OAUTH_CLIENT_SECRET
  • SMOKE_AUDIT_ID

The smoke test derives the API host (both for OAuth and for SDK calls) from the SDK's own ServerList array, so there is no SMOKE_API_BASE_URL secret — keeping it out avoids the class of bug where the OAuth host and the SDK host drift apart.

The OAuth client secret is one-time-display; if it's lost, the only path is to create a new OAuth client in the demo account and update the secret.

Production-traffic considerations

Smoke tests issue real requests to api.vanta.com on every regen PR. To keep this safe:

  • The test code must never pass a debugLogger option to the Vanta constructor. It dumps request/response detail to stdout, which would appear in CI logs.
  • The OAuth response body is suppressed from any thrown errors — only the status code and the RFC-6749 error / error_description fields are reported. The raw body can echo client_id.
  • v1 of the smoke test is read-only. Mutation calls (createAuditor, createCustomControl, createCommentForEvidence, etc.) are explicitly out of scope until we add deterministic create→assert→delete sequencing.

Empty fixture data

The fixture audit currently has no IRs, evidence, comments, snapshots, or issues. Endpoints that drill into these (e.g. getEvidenceUrls after listEvidence) use Vitest's ctx.skip(...) to report as skipped rather than failed when the parent list is empty.

This means a green-but-yellow CI run is the expected state today: SDK health checks pass, fixture-coverage gaps show up as skips. A regression in an endpoint that does have data (e.g. listAudits) still fails red.

Running locally

export SMOKE_OAUTH_CLIENT_ID=...
export SMOKE_OAUTH_CLIENT_SECRET=...
export SMOKE_AUDIT_ID=6a01f08107cb79d9ad9645cb

# Build the SDK so the smoke test can import compiled .js output via file:..
npm ci
npm run build

# Install smoke-test deps (links the parent SDK in) and run
cd smoke-test
npm install
npm test

The smoke-test directory is a standalone npm package that pulls the SDK in via "vanta-auditor-api-sdk": "file:.." — not a workspace of the parent package. This is deliberate: the parent's package.json is regenerated by Speakeasy on every regen, so adding the smoke-test as an npm workspace there would silently disappear on the next cron run. See smoke-test/package.json for the file:.. dependency that wires the local SDK in.

Without the env vars, every test reports as skipped (via Vitest's test.skipIf). This is by design so npm test doesn't break for engineers who don't have demo-account creds.

CI behavior

The Smoke Test workflow runs on every pull_request targeting main and on workflow_dispatch. It's a required status check on main — non-passing smoke means the regen PR doesn't merge.

Bypass: skip-smoke label

A maintainer may apply the skip-smoke label on a PR to bypass the gate when the failure is upstream (e.g. a Vanta API regression being fixed elsewhere) and the SDK changes themselves are fine. The workflow still runs and reports passing, but logs that it bypassed.

Bypass policy:

  • Link the tracking issue (Vanta-side bug / Speakeasy issue / etc.) in the PR description before applying the label.
  • Don't apply the label to ship a known-broken SDK. The whole point of the gate is to keep broken SDKs out of npm.

Recovery runbooks

Smoke test fails with fixture audit 6a01...cb not found

The fixture audit was likely deleted. To recreate:

  1. Log into the demo audit-firm account (1Password).
  2. Create a new SOC 2 Type II audit against Amalgamated Widgets.
  3. Update the SMOKE_AUDIT_ID GitHub Actions secret in both SDK repos to the new audit's ID.
  4. Re-run the regen PR.

Smoke test fails at OAuth (returned HTTP 401)

OAuth client credentials were rotated or revoked. To rotate:

  1. In the demo audit-firm account's Developer Console, generate a new client_credentials OAuth client.
  2. Capture the client_id and (one-time-display) client_secret.
  3. Update SMOKE_OAUTH_CLIENT_ID and SMOKE_OAUTH_CLIENT_SECRET GH secrets in both SDK repos.
  4. Re-run the regen PR.

Smoke test fails on ServerList contains Commercial

The Speakeasy overlay either lost the Commercial host or generated something with a different prefix. Inspect .speakeasy/speakeasy-modifications-overlay.yaml and the regenerated src/lib/config.ts ServerList array. This is the kind of bug the smoke test exists to catch — investigate before bypassing.