optimize-anything

LLM-guided optimization for text artifacts using an iterative propose-evaluate-reflect loop with a bring-your-own evaluator.

Quickstart (v2)

# 1) Install
curl -fsSL https://raw.githubusercontent.com/ASRagab/optimize-anything/main/install.sh | bash

# 2) Create a seed artifact
echo "Write a concise support prompt" > seed.txt

# 3) Optimize with the built-in LLM judge
optimize-anything optimize seed.txt \
  --judge-model openai/gpt-5.6-luna \
  --objective "Improve clarity and specificity" \
  --model openai/gpt-5.6-sol \
  --budget 20 \
  --workers 4 \
  --cache \
  --run-dir runs \
  --output result.txt

CLI stdout returns a JSON summary — see Result Contract for the full shape.

Model defaults

The proposer defaults to openai/gpt-5.6-sol after checking --model and OPTIMIZE_ANYTHING_MODEL. Generated judge and composite evaluators default to openai/gpt-5.6-luna. LLM judge calls use each provider's sampling defaults unless an explicit temperature is supplied.

LLM API authentication

Export the API key for every provider selected by --model, --judge-model, or --providers before launching the CLI or plugin host:

export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
export GEMINI_API_KEY="..."
export OPENROUTER_API_KEY="..."

The default proposer and judge require OPENAI_API_KEY. Multi-provider validation requires each selected provider's key; Gemini also accepts GOOGLE_API_KEY. Claude Code or Codex login is separate from provider API authentication. Plugin installation does not store credentials, so export keys before launching the host and restart it after changes. --api-base changes the endpoint but does not select a provider or its credentials. Do not commit API keys or .env files.

How It Works

optimize-anything runs a GEPA (Guided Evolutionary Prompt Algorithm) loop: propose → evaluate → reflect, repeating until budget is exhausted or early stopping kicks in.

seed.txt ──► [Propose] ──► candidates
                 ▲               │
                 │           [Evaluate]
             [Reflect] ◄──── scores + diagnostics

1. Propose — The optimizer generates candidate artifacts from your seed (or from scratch in seedless mode).
2. Evaluate — Each candidate is scored by your evaluator. Three evaluator types are supported: a command evaluator (any executable that reads JSON on stdin and writes a score on stdout), an HTTP evaluator (a service that accepts POST requests), or a built-in LLM judge (no evaluator script required — just pass --judge-model).
3. Reflect — Scores and diagnostics feed back into the next proposal round. The loop continues, progressively improving the artifact toward your objective.

The evaluator is the only thing you bring. Everything else — proposal strategy, reflection, early stopping, caching, parallelism — is handled by the optimizer.

Runtime Modes

Dataset / Valset modes

Use --dataset for multi-task optimization (one evaluator call per example). Add --valset for generalization validation.

optimize-anything optimize prompt.txt \
  --judge-model openai/gpt-5.6-luna \
  --objective "Generalize across customer request types" \
  --dataset data/train.jsonl \
  --valset data/val.jsonl \
  --model openai/gpt-5.6-sol \
  --budget 120 --workers 6 --cache --run-dir runs

Multi-provider validation

Cross-check one artifact with multiple judge providers:

optimize-anything validate result.txt \
  --providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash \
  --objective "Score clarity, constraints, and robustness" \
  --intake-file intake.json

Gemini uses LiteLLM's gemini/ provider prefix.

Seedless mode

No seed file required; GEPA bootstraps from objective.

optimize-anything optimize --no-seed \
  --objective "Draft a concise, testable API prompt" \
  --model openai/gpt-5.6-sol \
  --judge-model openai/gpt-5.6-luna

--no-seed requires both --objective and --model.

Early stopping and cache reuse

• Early stop is auto-enabled when --budget > 30 (or force with --early-stop)
• Reuse prior evaluator cache with --cache-from (requires --cache + --run-dir)
• Evaluator calls run in parallel by default; pass --no-parallel for evaluators that write shared temp files, depend on process-global state, or need strict provider rate-limit control.
• --proposals-per-iteration N asks GEPA to generate N mutations from the selected parent each iteration. It is independent of --workers, which only controls concurrent evaluator calls, and defaults to one proposal.

Proposal fan-out increases reflection and evaluation work. Small training sets may reuse the same minibatch across proposals, and GEPA checks the metric-call budget between iterations, so the final iteration can exceed --budget after its work has been scheduled.

GEPA 0.1.4 can migrate older run state forward. State written by 0.1.4 is not expected to load under 0.1.1 after a rollback; --cache-from remains limited to copying the prior fitness cache.

optimize-anything optimize seed.txt \
  --evaluator-command bash eval.sh \
  --model openai/gpt-5.6-sol \
  --budget 150 \
  --cache --cache-from runs/run-20260303-120000 \
  --run-dir runs \
  --early-stop --early-stop-window 12 --early-stop-threshold 0.003

Score range options

For command/HTTP evaluators:

• --score-range unit (default): enforce score in [0, 1]
• --score-range any: allow any finite float

optimize-anything optimize seed.txt \
  --evaluator-command bash eval.sh \
  --model openai/gpt-5.6-sol \
  --score-range any

Optimization observability benchmark

Use the maintainer benchmark when you want evidence that an optimization loop improved a useful artifact and was not just a seed-only or scorer-gaming run.

uv run python scripts/optimization_observer.py setup
uv run python scripts/optimization_observer.py report \
  --run-dir integration_runs/optimization-observability/evaluator-generation-guidance/runs/run-YYYYMMDD-HHMMSS \
  --benchmark examples/optimization-observability/evaluator-generation-benchmark.json \
  --strict

See examples/optimization-observability/README.md for the full benchmark commands, expected report fields, acceptance criteria, and troubleshooting.

CLI Subcommands

• optimize
• generate-evaluator
• intake
• explain
• budget
• score
• analyze
• validate

Agent Plugins

The Claude Code plugin and Codex plugin share the same skills/ tree and bundled locked runtime. Plugin users need uv and Python 3.10 or newer, but do not need a global optimize-anything command. The standalone CLI remains a separate installation choice.

Prompt Optimization Workflow

Claude Code users invoke $optimize-prompt; Codex users invoke the namespaced $optimize-anything:optimize-prompt. Both accept prompt text, a standalone file, an embedded prompt region, or a list of independent prompt files.

Evidence mode	Evaluation	What it proves
Fast mode	Scores the prompt text for clarity, constraints, and task fitness	Prompt-quality evidence only
Rigorous mode	Runs the candidate on representative inputs, then scores task outputs	Task-performance evidence for the tested examples
Composite mode	Runs deterministic hard constraints before the rigorous judge	No subjective score can override a failed gate

Rigorous datasets use JSONL records with input, optional expected, criteria, and hard_constraints. See prompt-execution-dataset.md for representative examples, the default system-prompt adapter, custom adapter guidance, and expected cost controls. Use explicit proposer, target, and judge models; bound calls with a small dataset, budget, and early stopping.

The workflow captures the baseline and writes search output to a temporary or run directory. It accepts only a positive comparable score delta with all hard constraints satisfied, plus required held-out acceptance in rigorous mode. Independent prompt files receive separate decisions. Coupled components require an explicit structured-candidate adapter and are not optimized as unrelated files.

Inline example:

$optimize-prompt Improve this prompt in fast mode and return the accepted result:
"Summarize this."

The response includes the complete accepted prompt, evidence mode, and score delta; it does not write a repository file.

Embedded repository example:

$optimize-prompt Optimize SYSTEM_PROMPT in src/agent.py with the examples in
evals/prompt.jsonl. Apply it only if held-out acceptance passes.

The workflow optimizes a captured copy, replaces only SYSTEM_PROMPT, preserves the source representation, and runs the cheapest relevant parse or targeted test. Rejected candidates leave the file unchanged.

Plugin Regression Workflow

Use the regression harness when you want to verify that Claude can actually invoke the plugin correctly end-to-end, not merely that the CLI itself still works.

# Direct plugin regression run
uv run python scripts/plugin_regression.py

# Full repo validation including plugin regression
uv run python scripts/check.py --with-plugin

Requirements:

• claude CLI installed and authenticated
• OPENAI_API_KEY set in the shell that launches the command
• ANTHROPIC_API_KEY for validate or the full scenario set

The harness runs existing CLI scenarios plus bounded prompt inline-return and repository-apply scenarios. --dry-run verifies prompt and artifact wiring without credentials or model calls.

Claude Code Plugin

/plugin marketplace add ASRagab/optimize-anything
/plugin install optimize-anything@optimize-anything

For a local clone, replace the repository name in the first command with its absolute path. Restart Claude Code after installation or update, then invoke $optimize-prompt or an existing slash command.

Codex Plugin

codex plugin marketplace add ASRagab/optimize-anything
codex plugin add optimize-anything@optimize-anything

For local development, pass the clone path to codex plugin marketplace add. Start a new Codex thread after install or update, then invoke $optimize-anything:optimize-prompt so skill discovery refreshes. See install.md for update, removal, verification, and the optional standalone CLI path.

Slash Commands

Command	Description
/optimize-anything:optimize	Guided optimization workflow — walks you through mode selection, evaluator setup, execution, and results
/optimize-anything:quick	Zero-config one-shot optimization. Just provide a file and objective.
/optimize-anything:analyze	Discover quality dimensions for an artifact and objective
/optimize-anything:score	Score a single artifact without optimization
/optimize-anything:validate	Cross-validate with multiple LLM judges
/optimize-anything:compare	Side-by-side comparison of two artifacts
/optimize-anything:budget	Get a budget recommendation for your artifact
/optimize-anything:explain	Preview the optimization plan without running it
/optimize-anything:intake	Normalize and validate an intake specification

Skills

Both plugins include four skills that the host can invoke:

• optimize-prompt — Build the rubric, choose fast or rigorous evidence, optimize outside the source, and return or safely apply an accepted prompt
• optimization-guide — Full workflow walkthrough covering modes, configuration, budget, and result interpretation
• generate-evaluator — Choose the right evaluator pattern (judge, command, composite) and generate a script
• evaluator-patterns — Library of ready-to-use evaluator templates for prompts, code, docs, and agent instructions

Typical Plugin Workflow

/optimize-anything:analyze prompt.txt --objective "Score clarity"
  → discovers quality dimensions

/optimize-anything:quick prompt.txt "improve clarity and specificity"
  → runs analyze + optimize with sensible defaults, shows diff

/optimize-anything:validate result.txt --providers openai/gpt-5.6-luna anthropic/claude-sonnet-5 gemini/gemini-3.6-flash
  → cross-checks the result with multiple judges

Reference

Result Contract

CLI stdout returns a JSON summary with these fields:

• best_artifact
• total_metric_calls
• score_summary (initial, latest, best, deltas, num_candidates)
• top_diagnostics (list of {name, value})
• plateau_detected, plateau_guidance
• optional evaluator_failure_signal
• optional early_stopped, stopped_at_iteration
• budget_utilization (requested, evaluator_calls, candidates_accepted, efficiency)

Evaluator Protocol (v2)

Evaluator input payload (stdin JSON for command mode, POST JSON for HTTP mode):

{"_protocol_version": 2, "candidate": "...", "example": {...}, "task_model": "..."}

• candidate is required
• _protocol_version, example, and task_model are optional/additive
• legacy evaluators that only read candidate remain compatible

Evaluator output payload:

{"score": 0.75, "notes": "optional diagnostics"}

• score is required
• additional keys are treated as side-info

Intake

Intake is optional structured guidance you pass to the optimizer to shape how evaluation works. Instead of relying solely on an --objective string, intake lets you declare quality dimensions, hard constraints, evaluation patterns, and execution preferences — giving you finer control over what "better" means for your artifact.

Use intake when your evaluation criteria are multi-dimensional, when you need to enforce hard constraints, or when you want consistent evaluation behavior across runs.

Pass it inline or from a file:

# Inline
optimize-anything optimize seed.txt \
  --intake-json '{"quality_dimensions": ["clarity", "specificity"], "hard_constraints": ["max 100 words"]}' \
  --judge-model openai/gpt-5.6-luna

# From file
optimize-anything optimize seed.txt \
  --intake-file intake.json \
  --judge-model openai/gpt-5.6-luna

optimize-anything intake normalizes and validates these keys:

• artifact_class
• quality_dimensions
• hard_constraints
• evaluation_pattern
• execution_mode
• evaluator_cwd

optimize flags (complete)

Exactly one evaluator source is required: --evaluator-command OR --evaluator-url OR --judge-model.

Flag	Description	Default
--no-seed	Run without seed file; bootstrap from objective	false
--evaluator-command <cmd...>	Command evaluator (stdin/stdout JSON)	--
--evaluator-url <url>	HTTP evaluator endpoint	--
--intake-json <json>	Inline intake spec	--
--intake-file <path>	Intake spec file	--
--evaluator-cwd <path>	Working dir for command evaluator	--
--objective <text>	Optimization objective	--
--background <text>	Extra domain context	--
--dataset <train.jsonl>	Training dataset JSONL	--
--valset <val.jsonl>	Validation dataset JSONL (requires --dataset)	--
--budget <int>	Max evaluator calls	100
--output, -o <file>	Write best artifact to file	--
--model <model>	Proposer model (or env fallback)	OPTIMIZE_ANYTHING_MODEL, then openai/gpt-5.6-sol
--judge-model <model>	Built-in LLM judge evaluator model	--
--judge-objective <text>	Judge objective override	falls back to --objective
--api-base <url>	Override LiteLLM API base	--
--diff	Print unified diff (seed vs best) to stderr	false
--run-dir <path>	Save run artifacts in timestamped run dir	--
--parallel	Explicitly enable parallel evaluator calls	true
--no-parallel	Run evaluator calls serially	--
--workers <int>	Max workers for parallel evaluation	--
--proposals-per-iteration <int>	Candidate mutations per iteration; independent of evaluator workers	1
--cache	Enable evaluator cache	false
--cache-from <run-dir>	Copy prior fitness_cache into new run	--
--early-stop	Enable plateau early stop	auto on when budget > 30
--early-stop-window <int>	Plateau window size	10
--early-stop-threshold <float>	Min improvement required over window	0.005
--spec-file <path>	Load TOML spec defaults	--
--task-model <model>	Optional metadata forwarded to evaluators	--
--score-range unit|any	Score validation mode for cmd/http	unit

Learn More

• EXAMPLES.md
• WALKTHROUGH.md
• CONCEPTS.md
• PROTOCOL.md