Monitor non-interactive Python environment tooling after release

[eleanorjboyd]

Background

#25940 will change environment-tool behavior so auto-approved agent calls do not block on extension-owned UI. Because this is a user-facing behavior change in environment creation and selection, we need release monitoring that can detect regressions before broadening the contract or transferring ownership under #26078.

Define privacy-conscious measures

Where existing telemetry policy permits, measure aggregate outcomes such as:

• invocation and completion counts;
• success, structured failure, cancellation, and timeout rates;
• whether an agent invocation requested user-input UI;
• environment reuse versus creation;
• creation failure category;
• retries or repeated calls within a task/session;
• time to completion;
• active backend/capability tier;
• Python Environments enabled, disabled, or unavailable state.

Do not collect workspace paths, environment paths, package names, dependency contents, prompts, or other sensitive workspace details.

Rollout plan

• Establish a pre-change baseline where comparable data exists.
• Define dashboards or queries for the first pre-release and stable releases containing the fix.
• Compare completion, failure, cancellation, latency, retry, and unexpected-interaction rates before and after release.
• Review qualitative issue and feedback trends alongside telemetry.
• Define thresholds for pausing rollout, reverting behavior, or opening follow-up bugs.
• Record limitations where the platform cannot distinguish tool approval from extension-owned UI.

Acceptance criteria

• Required telemetry events or existing signals are documented and reviewed for privacy.
• Success and failure categories align with the structured outcomes implemented by Environment tools block auto-approved agent calls on user input #25940.
• A baseline, observation window, owner, and review cadence are defined before stable rollout.
• Explicit guardrail and rollback thresholds are recorded.
• Results are summarized back on Meta: Evolve Python environment tools for autonomous agent workflows #26078 before expanding or transferring the tool contract.