Skip to content

feat(cli): actionable errors for missing models and credentials (phase 1 of #3442)#3452

Open
Sayt-0 wants to merge 1 commit into
mainfrom
feat/3442-actionable-model-credential-errors
Open

feat(cli): actionable errors for missing models and credentials (phase 1 of #3442)#3452
Sayt-0 wants to merge 1 commit into
mainfrom
feat/3442-actionable-model-credential-errors

Conversation

@Sayt-0

@Sayt-0 Sayt-0 commented Jul 3, 2026

Copy link
Copy Markdown
Member

Part of #3442 (Phase 1: actionable errors). Phases 2-4 (doctor, guided setup, docs tutorial) are independently shippable and left to follow-up PRs, as laid out in the issue.

Changes mapped to the issue scenarios

# Scenario Before After
3 Bare docker agent, non-TTY, empty stdin (CI) exit 0, zero output error naming the next steps (pass a message, pipe one, or use a terminal), exit 1; piped prompts keep working
2 DMR endpoint reachable, docker model CLI broken, model not pulled TUI starts, first message fails with raw HTTP 404: POST .../chat/completions fails at client creation: "model X is not available in Docker Model Runner" + docker model pull X
1/2 "Not installed" detection exact string match on the full docker usage text content match on unknown flag: --json, shared between NewClient and ListModels
4 Missing model credentials (RequiredEnvError, first_available variant) "use one of the built-in environment variable providers" (never named) each secret source named with a one-line example (export, --env-from-file, keychain, pass, Docker Desktop, credential helper), docs URL, and the local-model alternative (only when the missing vars are model credentials)
6 run --env-from-file typo.env missing/malformed file logged via slog (invisible without --debug), run continues run aborts: --env-from-file: open typo.env: no such file or directory; parse errors name the file and expected format
5 Exec-mode failure printed twice (CLI printer ❌ ... + cobra Error: ...) printed once by cobra, on stderr

Sample output (missing model credentials)

Error: The following environment variables must be set:
 - ANTHROPIC_API_KEY

Provide them using any of these sources:
 - Shell environment:  export ANTHROPIC_API_KEY=<value>
 - Env file:           docker agent run --env-from-file <file> ...
 - macOS Keychain:     security add-generic-password -a "$USER" -s ANTHROPIC_API_KEY -w
 - pass:               pass insert ANTHROPIC_API_KEY
 - Docker Desktop:     keys stored in Docker Desktop are picked up when you are signed in
 - Credential helper:  set `credential_helper` in ~/.config/cagent/config.yaml

See https://docs.docker.com/ai/docker-agent/guides/secrets/ for details.

No API key? Run a local model instead: docker agent run --model dmr/ai/qwen3 ...
Models already pulled in Docker Model Runner are detected automatically by the default `auto` model.
(`docker agent models` lists the models available to you)

Implementation notes

docker agent run (dmr model, no BaseURL / MODEL_RUNNER_HOST)
  |- docker model status ok        -> inspect + pull prompt (unchanged)
  |- status fails, "unknown flag"  -> ErrNotInstalled (content match, was exact match)
  `- status fails otherwise        -> GET <dmr>/models at client creation
       |- model listed             -> proceed
       |- model absent             -> ModelNotAvailableError ("docker model pull X")
       `- endpoint unqueryable     -> "cannot query Docker Model Runner at <url>" + install link
  • ModelNotAvailableError passes through the teamloader unwrapped (same treatment as PullFailedError) and exposes a one-line summary when nested under AutoModelFallbackError, so guidance blocks do not stack.
  • The non-TTY check lives at input time in cli.Run rather than at bare-command dispatch: a dispatch-level isatty(stdout) check would break the working echo "prompt" | docker agent > out.txt flow. The acceptance criterion (actionable error, exit != 0, never a silent no-op) is met either way; if the dispatch-level check seems preferable, it can be changed.
  • --env-from-file validation reuses the provider chain built by RuntimeConfig.EnvProvider() (new EnvFilesError() accessor) and aborts in the shared command pre-run, so every command taking the flag is covered.
  • RequiredEnvError.MissingModelCredentials is computed in gatherMissingEnvVars so the local-model hint is not shown for missing tool secrets (e.g. GITHUB_PERSONAL_ACCESS_TOKEN).

Testing

Area Coverage
pkg/environment message content for both flag states, parse error names the file
pkg/model/provider/dmr errIndicatesNotInstalled, modelAvailable (tag/repo matching), checkModelAvailable via httptest (pulled / not pulled / unqueryable), NewClient shim test with variant docker usage text
pkg/cli empty stdin (bare and -) fails, piped stdin still works, ErrorEvent returned once and not printed
pkg/config EnvFilesError (missing, malformed, valid, clone), MissingModelCredentials wiring, updated first_available assertions
cmd/root pre-run aborts on missing and malformed --env-from-file

All scenarios also verified end to end against a built binary (including a live reproduction of scenario 2: reachable DMR, broken docker model CLI, not-pulled model). Existing e2e expectations (environment variables must be set, OPENAI_API_KEY on stderr) are preserved. Remaining suite failures (TestURLSource_Read_RejectsLocalAddresses, SSRF tests, TestLoadExamples DMR pull) reproduce on main and are environment-related.

Phase 1 of #3442:

- Bare `docker agent` with empty non-TTY stdin fails with next steps
  instead of exiting 0 silently; piped prompts keep working
- DMR: "not installed" detection no longer depends on the exact docker
  CLI usage text; when `docker model status` fails, model availability
  is verified through the DMR HTTP API at client creation so a
  not-pulled model reports "docker model pull X" instead of a raw
  HTTP 404 at message time
- RequiredEnvError and the first_available variant name every secret
  source with a one-line example, link to the secrets guide, and
  suggest the local-model alternative (including that pulled DMR
  models are picked up automatically by the `auto` model) when model
  credentials are missing
- --env-from-file with an unreadable or malformed file aborts the run
  instead of being logged and skipped; parse errors name the file
- Exec mode prints the final error once (cobra), not twice
@Sayt-0 Sayt-0 requested a review from a team as a code owner July 3, 2026 14:58
@aheritier aheritier added kind/feat PR adds a new feature (maps to feat:). Use on PRs only. area/cli CLI commands, flags, output formatting area/config For configuration parsing, YAML, environment variables area/providers/docker-model-runner Docker Model Runner (DMR) local inference labels Jul 3, 2026
@dgageot

dgageot commented Jul 3, 2026

Copy link
Copy Markdown
Member

@Sayt-0 we need to review some of the messages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli CLI commands, flags, output formatting area/config For configuration parsing, YAML, environment variables area/providers/docker-model-runner Docker Model Runner (DMR) local inference kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants