Add Pyodide support and CI jobs for Zarr - #1903
Open
agriyakhetarpal wants to merge 137 commits into
Open
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This was referenced May 23, 2024
Closed
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
agriyakhetarpal
force-pushed
the
emscripten
branch
from
May 27, 2024 20:13
f23d1d9 to
cdf0bb2
Compare
agriyakhetarpal
force-pushed
the
emscripten
branch
from
May 27, 2024 20:48
b52b6c4 to
7ae9a97
Compare
This is because they test async code and also use threads. See the following issues: pyodide/pyodide#2221 pyodide/pyodide#237
Author
|
Since we've now released Pyodide 314.0.0 stable, I've bumped to it and removed the TODO note. I've pinned pytest<9.1 due to a regression which is being tracked upstream: pytest-dev/pytest#14591, pytest-dev/pytest#14592. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds official support for running Zarr v3 via Pyodide, running in the browser via WebAssembly. I have introduced a CI job and various code changes to ensure compatibility, disable unsupported features, and adjust test infrastructure accordingly. The most important changes are grouped below.
I've provided a short summary below. AI disclosure: it was partially generated using the GitHub Copilot actions button that appears when editing the PR description. I've carefully read it and augmented it by adding some points of my own, and I've removed some points that were not particularly relevant.
WASM/Pyodide platform support
emscripten.ymlworkflow to run CI tests on Pyodide/WASM.IS_WASMinsrc/zarr/_constants.pyto detect WASM environments, and used it throughout the codebase for platform-specific logiczarr.core.syncto support WASM: synchronous APIs now check for JSPI availability. Thread-based event loops are disabled in WASM.codec_pipeline.max_workersresolves to 1 in WASM environments as we can't spawn new threads, soFusedCodecPipelinestays on its sequential path even if a larger value is set explicitly._safe_moveinzarr.storage._localhandles file moves safely on Emscripten, as inode-based hard links are not present in its virtual file systems. I've split it up into three platform-specific methods.Testing infrastructure and CI adjustments
pyproject.toml. Its pytest invocation passes-o faulthandler_timeout=0because pytest's faulthandler watchdog needs a thread to run its timer and cannot start one under Emscripten.asyncio.Runnerin WASM environments to integrate with Pyodide’s native event loop, the WebLoop. This fixes two issues: (i) variousResourceWarnings appearing with consecutive async tests as a result of interactions between thepytest-asyncioprivate event loop and the webloop, and (ii) the webloop takes over as the running loop and thus making pytest-asyncio run coroutines on it instead of trying to run on its own.threadingandmultiprocessing(see also: Threading support pyodide/pyodide#237).Here are some related conversations where Pyodide (and its related discussions) has come up as a topic: #536, #1018, #1019, #1104, fsspec/filesystem_spec#960, and zarr-developers/community#43
It is to be noted that this PR has been accompanied by a corresponding sister PR for Pyodide builds for
numcodecsat zarr-developers/numcodecs#529, wherenumcodecsbuilds successfully and the test suite passes. The change was proposed after discussion at one of the community meetings for Zarr on March 6th, 2024 (meeting notes).TODO:
Add docstrings and API docs for any new/modified user-facing classes and functionsNo user-facing features or public API were proposedNew/modified features documented inThere are no new features in this PRdocs/user-guide/*.rstchanges/Additional context
Here are a few xrefs to some other PRs that have been implemented for packages in the Scientific Python ecosystem circa 2024:
scikit-image): Build and test PyWavelets Pyodide wheels in CI PyWavelets/pywt#701 and Upload dev wheels to Anaconda.org + revamp wheels publishing workflow PyWavelets/pywt#714pandasrepository BLD, TST: Build and test Pyodide wheels forpandasin CI pandas-dev/pandas#57896scikit-imagescikit-image/scikit-image#7350As of May 2026, PEP 783 has been accepted, and wheels for the Pyodide platform, with a "pyemscripten" platform tag, are being made available via PyPI. This PR and the sister numcodecs PR I mentioned above are efforts made in this light to increase our adoption in the ecosystem. Our next release of Pyodide, v314.0.0, is coming soon, and two alphas have already been released. I can draw on the changes I've made in this PR to re-enable Zarr support after it was disabled due to the v3 upgrade in version 0.28 (pyodide/pyodide-recipes#99).
My previous (and still current, but slightly lower priority than before) long-term goal towards this change, when I opened this PR back in 2024, was to implement Sphinx-based interactive documentation via JupyterLite and WASM-powered in-browser kernels, as referenced in Quansight-Labs/czi-scientific-python-mgmt#19; see also Quansight-Labs/czi-scientific-python-mgmt#18. A pilot of this that can be readily tried out is available for the "API reference" pages under the PyWavelets documentation. This can be preceded by integrating Sphinx extensions such as
jupyterlite-sphinxfor hosted documentation. Nightly wheels from the Anaconda Scientific Python Nightly Wheels channel can be used for nightly/dev documentation, and hopefully pure Python wheels from PyPI with this PR included for stable versions.