Skip to content

Improve CLI stage telemetry and error reporting#807

Draft
zahlekhan wants to merge 2 commits into
thesysdev:mainfrom
zahlekhan:codex/improve-cli-stage-telemetry
Draft

Improve CLI stage telemetry and error reporting#807
zahlekhan wants to merge 2 commits into
thesysdev:mainfrom
zahlekhan:codex/improve-cli-stage-telemetry

Conversation

@zahlekhan

@zahlekhan zahlekhan commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a canonical cli_create_stage lifecycle event with stage rank, status, duration, and per-run/per-attempt IDs
  • report privacy-safe failure stage, category, code, exit status, signal, cancellation exit code, and HTTP status
  • classify pnpm/npm/yarn/network/auth failures while continuing to stream package-manager output to the terminal
  • distinguish successful, failed, skipped, and cancelled stages so auth skip, blank manual auth, --no-install, and unrequested skill installation do not reduce completion rates
  • preserve OAuth error causes and classify OIDC/key-mint HTTP, DNS, and timeout failures
  • terminate the package-manager process tree on Windows cancellation and retain only safe, allowlisted failure fingerprints from command output
  • keep the existing create funnel events compatible

Why

The current CLI reports that dependency installation or creation failed, but discards the process error and the existing CreateError.stage. Stage instrumentation is also manually duplicated, so fields and semantics drift between steps. This makes pnpm failures difficult to diagnose and blends intentional skips or retries into apparent drop-off.

This combines the canonical stage runner with hardened subprocess and error handling. Raw stdout/stderr, local paths, response bodies, project names, and user-entered values are never sent to PostHog. A bounded diagnostic window remains local, while only code-defined failure fingerprints are retained for classification.

Impact

PostHog can benchmark conversion and p50/p95 duration by stage and CLI version, and failures can be broken down by normalized category, package manager, package-manager version, Node version, auth method, and AI setup. Prompt cancellation, auth skip, --no-install, recoverable auth failure, and an unrequested skill install are reported separately.

Package-manager selection and install flags are unchanged: pnpm still runs pnpm install; this change does not add --strict-peer-dependencies.

Validation

  • 48 CLI tests across 8 files
  • pnpm --filter @openuidev/cli test
  • pnpm exec tsc -p packages/openui-cli/tsconfig.test.json --noEmit
  • pnpm --filter @openuidev/cli run build
  • pnpm --filter @openuidev/cli run lint:check
  • pnpm --filter @openuidev/cli run format:check
  • cloud create smoke test with --auth skip --no-install --no-skill --no-interactive

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant