On Windows, creating a multiprocessing pool leaks handles

[vstinner]

Example:

import ctypes.wintypes
import gc
import multiprocessing.pool

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

kernel32.GetCurrentProcess.argtypes = ()
kernel32.GetCurrentProcess.restype = ctypes.wintypes.HANDLE

kernel32.GetProcessHandleCount.argtypes = (ctypes.wintypes.HANDLE, ctypes.wintypes.LPDWORD)
kernel32.GetProcessHandleCount.restype = ctypes.wintypes.BOOL

def handle_count():
    # Pseudo-handle that doesn't need to be closed
    hproc = kernel32.GetCurrentProcess()

    handle_count = ctypes.wintypes.DWORD()
    if not kernel32.GetProcessHandleCount(hproc, ctypes.byref(handle_count)):
        raise ctypes.WinError(ctypes.get_last_error())

    return handle_count.value

def func():
    pool = multiprocessing.pool.Pool(1)
    pool.terminate()
    pool.join()
    pool.close()
    pool = None
    gc.collect()

def main():
    print("Initial handle count:", handle_count())
    for _ in range(4):
        func()
        print("Handle count:", handle_count())

if __name__ == "__main__":
    main()

Output with Python 3.16 on Windows:

Initial handle count: 116
Handle count: 123
Handle count: 127
Handle count: 131
Handle count: 135

I discovered the issue while working on PR gh-154140 which detects leaks of Windows handles in the Python test suite.

Linked PRs

• gh-154208: Fix Windows handle leak when a child process dies early #154365

[vstinner]

See also the old issue gh-82444 but it seems to be unrelated.

[vstinner]

Using https://learn.microsoft.com/en-us/sysinternals/downloads/handle program, I can see that each loop iterations leaks 4 NamedPipe handles. Example:

  258: File  (---)   \Device\NamedPipe\pyc-11440-3-ac3e556c28fc4736
  25C: File  (---)   \Device\NamedPipe\pyc-11440-3-ac3e556c28fc4736
  260: File  (---)   \Device\NamedPipe\pyc-11440-4-1fe407d4fe79b646
  264: File  (---)   \Device\NamedPipe\pyc-11440-4-1fe407d4fe79b646

[vstinner]

Ok, the problem is that the pool child processes are killed before they can run spawn_main().

The normal code, when the child process is not killed early:

• The main process (pool manager) duplicates handles and pass handles to child processes.
• Child processes close these handles by calling DuplicateHandle() with DUPLICATE_CLOSE_SOURCE (call DupHandle.detatch() of multiprocessing.reduction if I follow correctly).

This issue is similar to issue gh-87134: "[Windows] multiprocessing handle leak when child process is killed during startup/unpickling".

[vstinner]

See also similar issue #114720.