Test Explorer: Partial Representation of Complete Test Discovery

[dylan-at-na]

Hi folks,

I have test discovery working successfully for pytest, here's a snippet from the output when I refresh tests in the Testing panel...

2024-02-02 10:42:35.576 [info] Discover tests for workspace name: <repo> - uri: /u/<user>/repos/<repo>
2024-02-02 10:42:35.576 [info] Running discovery for pytest using the new test adapter.
2024-02-02 10:42:35.577 [info] Discover tests for workspace name: cvs_qa - uri: <path1>
2024-02-02 10:42:35.577 [info] Running discovery for pytest using the new test adapter.
2024-02-02 10:42:35.577 [info] All environment variables set for pytest discovery for workspace /u/<user>/repos/<repo>: <env vars>

2024-02-02 10:42:35.594 [info] > ~/repos/<repo>/.venv/bin/python -m pytest -p vscode_pytest --collect-only suites
2024-02-02 10:42:35.594 [info] cwd: ~/repos/<repo>
2024-02-02 10:42:35.791 [info] > ~/.pyenv/versions/3.10.13/bin/python -m pytest -p vscode_pytest --collect-only
2024-02-02 10:42:35.791 [info] cwd: <path1>
2024-02-02 10:42:36.310 [info] ============================= test session starts ==============================
platform linux -- Python 3.10.13, pytest-7.4.0, pluggy-1.2.0

2024-02-02 10:42:36.310 [info] rootdir: <path1>

2024-02-02 10:42:36.369 [info] ============================= test session starts ==============================
platform linux -- Python 3.10.4, pytest-7.4.3, pluggy-1.3.0

2024-02-02 10:42:36.372 [info] rootdir: /u/<user>/repos/<repo>
configfile: pyproject.toml
plugins: xdist-3.4.0

2024-02-02 10:42:37.232 [info] collected 191 items

2024-02-02 10:42:37.233 [info] 
<Package suites>
  ...
  ...
  ...
  <Module test_<suite1>.py>
    <Class TestFpu>
      <Function test_create[TCASE-30136-vs0-expected0]>
      <Function test_create[TCASE-30137-vs1-expected1]>
      <Function test_create[TCASE-30138-vs2-expected2]>
      <Function test_create[TCASE-30139-vs3-expected3]>
      <Function test_create[TCASE-30140-vs4-expected4]>
      <Function test_create[TCASE-30141-vs5-expected5]>
      <Function test_create[TCASE-30142-vs6-expected6]>
      <Function test_create[TCASE-30143-vs7-expected7]>

2024-02-02 10:42:37.233 [info]       <Function test_create[TCASE-30144-vs8-expected8]>
      <Function test_create[TCASE-30145-vs9-expected9]>
      <Function test_create[TCASE-30146-vs10-expected10]>
      <Function test_create[TCASE-30147-vs11-expected11]>
      <Function test_create[TCASE-30148-vs12-expected12]>
      <Function test_create[TCASE-30149-vs13-expected13]>
      <Function test_create[TCASE-30150-vs14-expected14]>
      <Function test_create[TCASE-30151-vs15-expected15]>
      <Function test_create[TCASE-30152-vs16-expected16]>
      <Function test_create[TCASE-30153-vs17-expected17]>
      <Function test_create[TCASE-30154-vs18-expected18]>
      <Function test_create[TCASE-30155-vs19-expected19]>
      <Function test_create[TCASE-30156-vs20-expected20]>
      <Function test_create[TCASE-30157-vs21-expected21]>
      <Function test_create[TCASE-30158-vs22-expected22]>
      <Function test_create[TCASE-30159-vs23-expected23]>
      <Function test_create[TCASE-30160-vs24-expected24]>
      <Function test_create[TCASE-30161-vs25-expected25]>
2024-02-02 10:42:37.233 [info] 
      <Function test_create[TCASE-30162-vs26-expected26]>
      <Function test_create[TCASE-30163-vs27-expected27]>
      <Function test_create[TCASE-30164-vs28-expected28]>
      <Function test_create[TCASE-30165-vs29-expected29]>
      <Function test_create[TCASE-30166-vs30-expected30]>
      <Function test_create[TCASE-30167-vs31-expected31]>
      <Function test_create[TCASE-30168-vs32-expected32]>
      <Function test_create[TCASE-30169-vs33-expected33]>
      <Function test_create[TCASE-30170-vs34-expected34]>
      <Function test_create[TCASE-30171-vs35-expected35]>

2024-02-02 10:42:37.233 [info]       <Function test_create[TCASE-30172-vs36-expected36]>
      <Function test_create[TCASE-30173-vs37-expected37]>
      <Function test_create[TCASE-30174-vs38-expected38]>
      <Function test_create[TCASE-30175-vs39-expected39]>
      <Function test_create[TCASE-30176-vs40-expected40]>
      <Function test_create[TCASE-30177-vs41-expected41]>
      <Function test_create[TCASE-30178-vs42-expected42]>
      <Function test_create[TCASE-30179-vs43-expected43]>
      <Function test_create[TCASE-31161-vs44-expected44]>

2024-02-02 10:42:37.233 [info]       <Function test_delete_v[TCASE-30180-vs0-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30181-vs1-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30182-vs2-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30183-vs3-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30184-vs4-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30185-vs5-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30204-vs6-vol1-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30205-vs7-vol2-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-30206-vs8-vol2-ResourceNotFoundError]>
2024-02-02 10:42:37.233 [info] 
      <Function test_delete_v[TCASE-30207-vs9-vol3-ResourceNotFoundError]>
      <Function test_delete_v[TCASE-31160-vs10-vol4-ResourceNotFoundError]>
      <Function test_recreate_v[TCASE-30186-vs0-target0-expected0]>
      <Function test_recreate_v[TCASE-30187-vs1-target1-expected1]>
      <Function test_recreate_v[TCASE-30188-vs2-target2-expected2]>
      <Function test_recreate_v[TCASE-30189-vs3-target3-expected3]>
      <Function test_recreate_v[TCASE-30190-vs4-target4-expected4]>
      <Function test_recreate_v[TCASE-30191-vs5-target5-expected5]>

2024-02-02 10:42:37.234 [info]       <Function test_recreate_v[TCASE-30192-vs6-target6-expected6]>
      <Function test_recreate_v[TCASE-30193-vs7-target7-expected7]>
      <Function test_recreate_v[TCASE-30194-vs8-target8-expected8]>
      <Function test_recreate_v[TCASE-30195-vs9-target9-expected9]>
      <Function test_recreate_v[TCASE-30196-vs10-target10-expected10]>
      <Function test_recreate_v[TCASE-30197-vs11-target11-expected11]>
      <Function test_recreate_v[TCASE-30198-vs12-target12-expected12]>
      <Function test_recreate_v[TCASE-30199-vs13-target13-expected13]>
      <Function test_recreate_v[TCASE-30200-vs14-target14-expected14]>
      <Function test_recreate_v[TCASE-30201-vs15-target15-expected15]>
      <Function test_recreate_v[TCASE-30202-vs16-target16-expected16]>
      <Function test_recreate_v[TCASE-30203-vs17-target17-expected17]>
      <Function test_recreate_v[TCASE-30208-vs18-target18-expected18]>
      <Function test_recreate_v[TCASE-30209-vs19-target19-expected19]>
      <Function test_recreate_v[TCASE-30210-vs20-target20-expected20]>
      <Function test_recreate_v[TCASE-30211-vs21-target21-expected21]>
      <Function test_recreate_v[TCASE-30212-vs22-target22-expected22]>
      <Function test_recreate_v[TCASE-30213-vs23-target23-expected23]>
2024-02-02 10:42:37.234 [info] 
      <Function test_recreate_v[TCASE-30214-vs24-target24-expected24]>
      <Function test_recreate_v[TCASE-30220-vs25-target25-expected25]>
      <Function test_api_change[TCASE-31464-old_vs0-vs_to_delete0-new_vs0-expected0]>
      <Function test_api_change[TCASE-31465-old_vs1-vs_to_delete1-new_vs1-expected1]>
      <Function test_api_change[TCASE-31466-old_vs2-vs_to_delete2-new_vs2-expected2]>
      <Function test_api_change[TCASE-31467-old_vs3-vs_to_delete3-new_vs3-expected3]>
      <Function test_api_change[TCASE-31468-old_vs4-vs_to_delete4-new_vs4-expected4]>
      <Function test_api_change[TCASE-31469-old_vs5-vs_to_delete5-new_vs5-expected5]>
      <Function test_api_change[TCASE-31470-old_vs6-vs_to_delete6-new_vs6-expected6]>
      <Function test_api_change[TCASE-31471-old_vs7-vs_to_delete7-new_vs7-expected7]>
      <Function test_api_change[TCASE-31472-old_vs8-vs_to_delete8-new_vs8-expected8]>
      <Function test_errors[TCASE-31162-vs0-IAZ]>

2024-02-02 10:42:37.245 [info] Test server connected to a client.
2024-02-02 10:42:37.277 [info] ResultResolver EOT received for discovery.
2024-02-02 10:42:37.277 [info] ResultResolver EOT received for discovery.
2024-02-02 10:42:37.277 [info] 
Logs path: /jenkins/workspace/artifacts/20240202104236345264
Logs files: ['tests_gw0.log']
========================= 191 tests collected in 0.90s =========================

2024-02-02 10:42:37.394 [info] Disposing data receiver for /u/<user>/repos/<repo> and deleting UUID; pytest discovery.
2024-02-02 10:42:38.454 [info] collected 0 items

2024-02-02 10:42:38.454 [info] Test server connected to a client.
2024-02-02 10:42:38.456 [info] ResultResolver EOT received for discovery.
2024-02-02 10:42:38.456 [info] 
========================= no tests collected in 2.15s ==========================

2024-02-02 10:42:38.486 [error] Subprocess exited unsuccessfully with exit code 5 and signal null on workspace <path1>.
2024-02-02 10:42:38.486 [info] Disposing data receiver for <path1> and deleting UUID; pytest discovery.

Unfortunately, these 191 tests don't appear in the panel. Focusing on just the module example provided above - only 1 test is added to the Test Explorer. See screenshot below...

Thats is the entire extent of discovered tests being reported for this module. But As we can see in the output above, there are significantly more tests present. Tests which vscode did discover but has chosen not to report. What could be the cause of this?

Can anybody advise on this?

[brettcannon]

This usually when you have a pytest plugin that changes test names or IDs. That possibly happening here?

[dylan-at-na]

You were right. We had the following piece of code in an attempt to make parameterised tests a little more identifiable.

def pytest_itemcollected(item):
    if hasattr(item._request.node, "callspec"):
        item._nodeid = item._request.node.callspec.params["id"]

Removing that has completely removed the issue around test discovery. Thank you so much!

[brettcannon]

You can try to specify a custom ID upfront via https://docs.pytest.org/en/8.0.x/reference/reference.html#pytest-param and we should pick that up.