Testing reliability & perf 2026 — workstream

[eleanorjboyd]

Tracking the user-reported testing regressions and performance complaints that emerged after the project-structure / [test-by-project] rework, plus the older follow-ups that ride alongside them.

The work is grouped into four problem areas. Each item below points at the existing user-filed issue (or a new engineering issue where there is no good match) so progress is visible in one place.

---

Problem areas (what users are reporting)

1. Test discovery is much slower than CLI pytest.
   Multiple users on large parametrized suites — 30k tests take 40s in the Test Explorer vs ~2s on the CLI; 328k tests take 66s vs ~10s. The O(n²) list scans and oversized discovery payload have been addressed; follow-up work remains on line-number/path compaction and validating the end-to-end improvement.

2. Test tree is rebuilt from scratch on every change.
   Saving any .py file re-discovers the whole workspace and wipes the existing tree. While re-discovery is in flight, users can't re-run or debug a test because the items have been cleared. "Debug: Restart" breaks for the same reason.

3. Run / debug pipeline regressions.
   Tests appear as "skipped" even though they ran; debug runs can lose results when the result pipe closes; pytest-subtests failures get reported as success; the env selected via the Python Environments API is not always honored.

4. Hard discovery failures still open.
   Smaller correctness bugs in the pytest plugin (HIDDEN_PARAM, pipe writer broken by mock.patch("builtins.open")).

---

Order of execution (impact × effort)

• Narrow python.testing.autoTestDiscoverOnSavePattern default so saving a non-test file does not trigger a full re-discovery. Default today is **/*.py; change it to conventional pytest/unittest filenames plus conftest.py, while preserving custom user patterns. Implementation and regression tests are complete locally; publication is pending. — python.testing.autoTestDiscoverOnSaveEnabled is running test discovery on change to any file (not just test files) #25866
• Replace list-membership dedup with dict/set in vscode_pytest. Removed the O(n²) scans that dominate heavily parametrized suites. — Extremely slow test discovery when using pytest for large test suite #25973, completed by Improve performance of pytest discovery for large test suites #25974
• Verify result-pipe drainage on cancellation. Implementing result pipe drainage on cancellation #25981 changed cancellation to allow buffered results to drain, but PyTest exits when subprocess finishes execution when ran through Testing UI #25872 remains open and still needs end-to-end verification against the reported debug/subprocess scenario.
• Surface an error payload when the env-extension subprocess exits non-zero. Legacy path already does this in pytestExecutionAdapter.ts; the env-extension path resolves the deferred silently. With no results and no error, every test defaults to skipped. — Python skip all Tests #25892
• Aggregate pytest-subtests reports before publishing the final test outcome. Multiple call-phase reports share one nodeid; the current collected_tests_so_far dedup makes the first report win, so a later failing subtest or final failure can be discarded. — failing pytest tests are marked as “success” when some of the subtests succeeded #25824
• Slim the pytest discovery JSON payload. Store shared path prefixes once and compact repeated absolute paths in test nodes. — [PERF] Pytest Discovery - highly parameterized tests lead to inefficient test node creation in vscode_pytest #25948, implemented by Enhance test discovery payload handling with compact/expansion on paths in payload #25982 with follow-up testing: refine compact discovery payload review feedback #26000. Remaining optimizations are tracked in open PRs optimize compact_path() in pytest discovery for the common case #26016 and optimize getting of line number in pytest discovery #26017; both currently require corrections before merge.
• Incremental tree updates in populateTestTree. Today processDiscovery does testItemIndex.clear() and populateTestTree always rebuilds. Diff old vs new test trees and only insert / remove / update changed items. Largest item; biggest user-visible fix; should land last. — After latest update: Testing tree of python tests reloads completely rather than just update #25822
• pytest.HIDDEN_PARAM discovery crash. process_parameterized_test does parent_part, parameterized_section = test_node["name"].split("[", 1), which raises ValueError when pytest emits a node id without [...] (i.e. when HIDDEN_PARAM is used). One-line guard. — pytest.HIDDEN_PARAM Causes Discovery Failure #25795
• Telemetry step 2 (Python-side). Add a meta block to the vscode_pytest discovery payload with subprocessDurationMs, pluginDurationMs, payloadBytes, parametrizedTestCount, and fold it into the new UNITTEST.DISCOVERY.DONE event so we can split slow discoveries into "subprocess vs plugin vs JS overhead". — new issue
• Make vscode_pytest pipe writer immune to mocked open. When user test code does mock.patch("builtins.open"), the pipe-writer open() call gets intercepted and serialization breaks (surfaces as unsupported operand type(s) for +=: 'int' and 'NoneType'). Capture open at import time before any test code can monkeypatch it. [vscode-pytest]: unsupported operand type(s) for +=: 'int' and 'NoneType' #25793 was closed as not planned after the information-request timeout; no fix landed. — new issue or reopen [vscode-pytest]: unsupported operand type(s) for +=: 'int' and 'NoneType' #25793
• Verify useEnvExtension() actually fires in VS Code 1.106+. Reporter pins a regression to that VS Code release. Telemetry-first: add an envSource field to the discovery/run events, look at the live data, then fix. — pytest discovery does not use specified environment #25718

---

Closing as already resolved

• Unittest test discovery fails for Python 3.15 #25802 — Unittest discovery on Python 3.15. User confirmed fixed in extension 2026.4.0 on May 9.
• Unwanted debug output when running unittests (again). #25807 — Unwanted unittest debug output. The offending print has already been removed from python_files/unittestadapter/execution.py.

---

How we measure progress

Instrumentation status:

• Add TS-side discovery/run timing and outcome telemetry. — Add testing performance telemetry #25980
• Send numeric timing/count fields through the telemetry measures channel so they are ingested. — Fix testing perf telemetry: send measurements via measures arg #26028
• Add Python-side phase timing and payload-size/count metadata.
• Document and run the analysis queries/dashboard slices used to validate each fix.

Each fix above has a corresponding metric so dashboards can verify the change actually moves the needle (and catch unintended regressions):

Area	Primary metric	What "fixed" looks like
Discovery perf	UNITTEST.DISCOVERY.DONE.totalDurationMs p50/p90 sliced by testCount bucket × mode	Large-suite p90 drops by an order of magnitude; mode='project' converges to mode='legacy'.
Tree rebuilt every save	UNITTEST.DISCOVERY.TRIGGER.fileKind='non-test' share; UNITTEST.TREE.UPDATE.rebuiltFromScratch share; msSinceLastTrigger p50	non-test share drops to ~0%; rebuiltFromScratch=false share grows as incremental updates land.
Run / debug pipeline	UNITTEST.RUN.DONE.missingCount > 0 share; pipeClosedEarly share; failureCategory distribution	missingCount>0 and pipeClosedEarly shares drop to near-0 on mode='project' and debugging=true.
Discovery hard failures	UNITTEST.DISCOVERY.DONE.failureCategory distribution	Each individual fix shrinks its corresponding bucket.

Per-area success criteria are checked off as each fix ships and the telemetry confirms the change.

---

Out of scope (deliberately, for now)

• Per-test or per-file names in telemetry — privacy-sensitive, not needed for the questions above.
• True added / removed counts in UNITTEST.TREE.UPDATE — needs an O(n) set diff per discovery; revisit only if beforeCount/afterCount + rebuiltFromScratch aren't enough signal.
• Migrating off named pipes entirely — out of scope for this workstream.

[flying-sheep]

I don’t know if calling the subtests feature pytest-subtests is accurate. I think pytest-subtests is the former plugin that used to provide this now-builtin feature. I think people care more about the builtin feature than a now-obsolete plugin.