Evaluate alternative Python environment tool surfaces for agent outcomes

[eleanorjboyd]

Background

The current configure_python_environment tool is a high-level orchestrator with a narrow input schema and potentially interactive behavior. Before committing to the richer contract ultimately owned by Python Environments, we should test whether different tool shapes and descriptions produce better agent outcomes.

This investigation is part of #26078 and should build on the current-versus-ideal flow map rather than assuming that exposing more arguments or more tools is automatically better.

Variants to compare

Evaluate representative versions of:

1. the current orchestration tool;
2. granular tools for discovery, creation, dependency installation, selection, and verification;
3. a richer declarative environment tool;
4. capability discovery combined with the compatibility tool;
5. a hybrid in which Python contributes a narrow compatibility tool and Python Environments contributes richer tools when active.

Schemas must remain honest: unsupported properties should fail explicitly and should not be silently ignored by a fallback backend.

Evaluation

Use representative agent tasks covering:

• creating a basic workspace virtual environment;
• honoring a Python version constraint;
• choosing among environment types;
• installing project dependencies;
• reusing an existing suitable environment;
• handling an existing .venv conflict;
• running with Python Environments disabled or unavailable;
• completing under tool auto-approval without user interaction.

For each variant, capture:

• autonomous task completion rate;
• invalid or unsupported tool calls;
• retries and loops;
• unexpected user-input requests;
• time/tool calls to completion;
• whether the created and selected environment actually matches the task;
• model confusion caused by overlapping or misleading tools.

Deliverable

Provide a recommendation for the long-term tool identities, ownership, schemas, descriptions, capability behavior, and migration path. Record tradeoffs and rejected alternatives.

Acceptance criteria

• Variants are compared using the same representative tasks and scoring criteria.
• Results distinguish schema quality from backend capability limitations.
• The recommendation addresses Python Environments enabled, disabled, and unavailable states.
• No recommended contract advertises behavior an active implementation may silently ignore.
• Findings and recommendation are linked from Meta: Evolve Python environment tools for autonomous agent workflows #26078.