geodms.pyd Python version matching across .m / .c installers + user-installed Python/GDAL

[MaartenHilferink]

Symptom

The Python binding geodms.pyd is linked against different Python ABIs in the two Windows installers:

Setup	geodms.pyd deps
.m (msbuild)	python3.dll, python313.dll
.c (cmake)	python312.dll

The .c setup bundles python312.dll next to geodms.pyd so the binding is self-contained. The .m setup ships no python*.dll — geodms.pyd only loads if a matching Python 3.13 runtime happens to be on the user's PATH or in the install dir.

Even when a runtime is found, the version-mismatch story for users who already have Python and/or GDAL pinned to a different version is unsolved:

• A user with Python 3.12 installed can use the .c binding but not the .m one.
• A user with QGIS / OSGeo4W (often ships its own Python and GDAL) may have conflicts on gdal.dll / proj.dll paths even when the Python version matches.
• geodms.pyd is also linked against the project's bundled gdal.dll — if the user's pre-existing Python loads a different GDAL first, behaviour is undefined.

What needs deciding

1. Pick one Python version per release line and align both Windows installers + Linux build. Pin via vcpkg manifest or a top-level CMake var.
2. Decide on the runtime-supply policy:
   • bundle pythonXYZ.dll next to geodms.pyd (self-contained, current .c behaviour); or
   • declare Python a hard prereq and document the exact version users must install; or
   • support multiple Python versions via per-version geodms_py3XY.pyd artifacts.
3. GDAL coexistence: geodms.pyd linked against bundled gdal.dll should not collide with a user's QGIS/OSGeo4W GDAL when both are on PATH. May need DLL search-order shim or rpath equivalent.
4. Linux: the .l build links against the system libpython3.so and libgdal.so via vcpkg — clarify the supported range, document in Installation Instructions wiki.

Workaround for now

.c installer's bundled python312.dll covers the standalone case.
.m installer + Python binding requires the user to have Python 3.13 (cpython.org installer) on PATH. Don't ship the .m Python binding to users who don't already have a matching runtime.

Discovered while comparing the v20.0.0 .m and .c installers (#release-v20.0.0).

[MaartenHilferink]

Some users also use python with a project that imports (a different version of) GDAL and then imports geodms.pyd, causing a gdal.dll conflict, even if they have the samen version of python as the one for which our bindings were made. So, to state it generic, a python user, say PBL, has a project, say GLOBIO, in which python 3.x is used. GLOBIO imports GDAL 3.y and wants to import GeoDMS.pyd of version z, which was built for/with python 3.alpha and uses and ships GDAL 3.beta. How can we and/or the user deal with this.

[MaartenHilferink]

Design summary: multi-Python × multi-GDAL build matrix

Discussion follow-up — separating the two failure modes and what a sustainable build pipeline looks like.

Two orthogonal mismatches

Axis	Failure mode	Decided by
Python ABI (e.g. 3.12 vs 3.13)	geodms.pyd won't import — wrong PyInit symbol	host python.exe at process start
GDAL DLL (e.g. 3.8 vs 3.11)	geodms.pyd imports, but at runtime calls the wrong gdal.dll and SEGVs	first Python package in the process to load gdal.dll wins

The GDAL one is the nastier — it survives "matching Python versions" and still crashes.

Python ABI — solved cheaply by multi-build

CMake's python3_add_library(... WITH_SOABI) already emits geodms.cp312-win_amd64.pyd, geodms.cp313-…, etc. — distinct filenames per ABI, no collision. Build matrix sets Python3_ROOT_DIR per loop iteration (cpython.org installers, side-by-side). All variants ship in one installer.

A small geodms/__init__.py does os.add_dll_directory(...) before import; Python's loader picks the SOABI-tagged .pyd matching sys.version_info automatically. This is exactly how PyPI wheels (numpy, GDAL, …) handle multi-version support.

Estimated cost: ~1 day to wire up + per-Python compile time in CI.

GDAL coexistence — separate installers per GDAL version

Don't try to fit multiple GDAL versions into one installer — gdal.dll, proj_9.dll, geos.dll names collide.

Naming pattern: GeoDms20.0.0.m.gdal308-Setup-x64.exe, GeoDms20.0.0.m.gdal311-…. The actual resolved version goes in the filename, not the word "latest" — "latest" is a moving target, the artifact has to be reproducible.

vcpkg mechanics

Two manifest files per GeoDMS version:

GeoDMS<ver>/
├── vcpkg.json                  # GDAL: track current
└── vcpkg.gdal-stable.json      # GDAL: pinned to 3.8.x

Stable manifest uses overrides + a baseline late enough to know about both versions:

{
  "builtin-baseline": "<vcpkg SHA where both 3.8 and 3.11 exist>",
  "overrides": [
    { "name": "gdal", "version": "3.8.5#1" },
    { "name": "proj", "version": "9.3.1"  },
    { "name": "geos", "version": "3.12.1" }
  ]
}

Co-pinning GDAL + PROJ + GEOS (and probably HDF5/libtiff/netcdf) is not optional — GDAL 3.8 was QA'd against older PROJ/GEOS. Figure the set out once, bake it into the manifest.

Manifest selection at build time: per-variant build dir with the right manifest copied to vcpkg.json. CMake presets:

windows-x64-release-gdal-current
windows-x64-release-gdal-stable

Binary caching is the sustainability lever

Without it, each manifest = ~3–5h port compilation on a CI box. Across N GeoDMS releases × 2 GDAL variants, untenable.

With vcpkg binary caching (NuGet feed on GitHub Packages is the natural fit for this org), each port build is hashed by version+features+triplet+compiler and archived. Subsequent builds download instead of compile. Across many GeoDMS releases sharing the same stable manifest, the gdal-stable dep cluster is built once.

Configure via VCPKG_BINARY_SOURCES. One-time setup, dramatic recurring savings.

Final release matrix per GeoDMS version

{gdal-current, gdal-stable} × {.m, .c}  →  4 installers
each installer bundles cp312, cp313, cp314 .pyd

User picks the .gdalXYZ matching what their other Python packages already load (osgeo, rasterio, OSGeo4W).

Per-release process

1. vcpkg.json — bump baseline, resolve, record GDAL version (e.g. 3.11.3).
2. vcpkg.gdal-stable.json — verify still resolves; advance baseline minimally if needed.
3. CI matrix builds 4 installers. Cache makes builds 2–4 fast.
4. Installer filenames carry the resolved GDAL version. Manifests committed at the release tag → fully reproducible from source years later.

Maintenance burden over time

Two failure modes for the stable pin:

1. GDAL 3.8 ages out of vcpkg. Mitigation: copy the port into ports-overlay-gdal-stable/gdal/ in this repo and add an overlay-ports entry. You now own that port.
2. Non-GDAL deps drift. Newer GeoDMS code may need a Boost feature not in the stable baseline. Advance the stable baseline minimally; add per-dep overrides; fall back to overlay ports.

Budget ~half a day per major GeoDMS release for stable-manifest hygiene. Most releases pass through with no changes thanks to binary cache hits.

Tradeoff

• Do the matrix: ~few days one-time setup (presets + manifests + binary cache + NSIS parameterization) + ~half a day per release. In return: PBL/GLOBIO-style users get a self-contained installer matching their existing GDAL, no surgery on their Python.
• Don't: pin one Python + one GDAL per release line, document it, push integration onto every user. Near-zero engineering cost, much higher user friction.

For a tool with institutional users on heterogeneous Python stacks, the matrix is worth it.

Concrete next steps (when ready to implement)

1. Pick the two GDAL targets and the Python set (e.g. cp312/cp313/cp314).
2. Author the two vcpkg.*.json manifests + verify co-pinned dep cluster compiles.
3. Add python/dll/__init__.py with os.add_dll_directory + SOABI dispatch.
4. CMake preset matrix + CI workflow.
5. Set up vcpkg binary cache (NuGet on GitHub Packages).
6. Parameterize .nsi with ${GdalVariant} + resolved version.
7. Document the "first-GDAL-loaded-wins" rule + "import geodms first" guidance in the wiki Installation Instructions.