ESPRESSIF'S IDF Python Wheels

This project automates the build and upload process of required Python Wheels by ESP-IDF. The wheels for multiple OSes and architectures are being built.

Supported architectures:

• Linux
  • Ubuntu - x86_64
  • ARMv7 - arm32
  • ARM64
• Windows - AMD64
• MacOS
  • x86_64
  • ARM64

Supported Python versions:

• 3.14
• 3.13
• 3.12
• 3.11
• 3.10
• 3.9
• 3.8

Note

This list of supported Python versions is automatically updated by update_python_versions.py script and update-python-versions.yml workflow.

For each release branch of ESP-IDF which is not EOL and ESP-IDF master branch, all the requirements and constraints files are automatically downloaded and wheels are built and uploaded.

Completely Automated

This repository has been completely automated. All the supported versions of ESP-IDF and Python versions are fetch and resolved automatically. The implementation of this logic is in the supported_versions.py script and get-supported-versions.yml workflow.

Also README.md file and pyproject.toml is automatically updated with the script update_python_versions.py and update-python-versions.yml workflow.

Supported Versions Action

This workflow is reusable action and it is possible to be called in other projects - it will generate supported_versions.json file with the following structure, which can be parsed and used in caller workflow to avoid developer interaction of changing the supported versions.

Also it sets the min_idf_major_version and min_idf_minor_version as a GitHub env variables so this can be used as well like this:

echo "MIN_IDF_MAJOR_VERSION=${{ needs.get-supported-versions.outputs.min_idf_major_version }}" >> $GITHUB_ENV

{
    "supported_idf": [
        "v5.5",
        "v5.4",
        "v5.3",
        "v5.2",
        "v5.1"
    ],
    "oldest_supported_idf": "v5.1",
    "supported_python": [
        "3.13",
        "3.12",
        "3.11",
        "3.10",
        "3.9",
        "3.8"
    ],
    "oldest_supported_python": "3.8"
}

Usage of Manual Wheels Build - DEFINED WHEELS WORKFLOW

If there is a need to manually build and upload wheels the defined-wheels workflow can be used for this. The pip package needs to be specified with marker support (e.g. coredump~=1.2;sys_platform!='win32') and check the architectures which should be wheels built and uploaded for. Multiple wheels can be separated by space.

Then the wheels are built and uploaded for all supported Python versions.

Requirements Lists

These lists are files for requirements that should be added or excluded from the main requirements list which is automatically assembled.

exclude_list.yaml

File for excluded Python packages in the main requirements list.

This YAML file is converted to Requirement from packaging.requirements because pip can handle this format, so the function for converting is designed to be compatible with PEP508 scheme. The opposite logic of exclude_list is handled by the function itself, which means it is supposed to be easy to use for developers, this is also the reason YAML format is used.

For every package_name there are options:

• version
  • supports all logic operators defined by PEP508 for versions (<, >, !=, etc.)
• platform
• python

which could be a string or a list of strings.

exclude_list template:

- package_name: '<name_of_package>'
    version: '<package_version_with_operator>' / ['<package_version_with_operator>', '<package_version_with_operator>']     # optional
    platform: '<platform>' / ['<platform>', '<platform>', '<platform>']                                                     # optional
    python: '<python_version_with_operator>' / ['<python_version>', '<python_version>', '<python_version>']                                                     # optional

The syntax can be converted into a sentence: "From assembled main requirements exclude package_name with version on platform for python version".

example:

- package_name: 'pyserial'
    version: ['>=3.3', '<3.6']
    platform: ['win32', 'linux', 'darwin']
    python: '>=3.9'

This would mean: "From assembled main requirements exclude pyserial with version >=3.3 and <3.6 on platform win32, linux, darwin for python version >=3.9".

From the example above is clear that the platform could be left out (because all main platforms are specified) so the options platform or version or python are optional, one of them or both can be not specified and the key can be erased. When only package_name is given the package will be excluded from main requirements.

PyPI Requires-Python preflight

Before running pip wheel, the build scripts can query the PyPI JSON API so that no release matching the requirement’s version specifier (including ==, ~=, and ranges such as >=x,<y) is installable on the current interpreter according to each candidate release’s Requires-Python metadata. In that case the requirement is skipped (with a log line) instead of invoking pip, which avoids noisy failures such as “No matching distribution found” when pip hides incompatible versions.

This is implemented in _helper_functions.py and wired into:

• build_wheels.py — after exclude_list is applied, for the main graph, include_list.yaml entries, and lines written to dependent_requirements.txt
• build_wheels_from_file.py — each requirement line from dependent_requirements.txt or from the CLI

If the project JSON cannot be fetched (network error, etc.), preflight does not skip; pip runs as usual.

This complements exclude_list.yaml: the YAML still expresses platform, markers, and build/repair policy where PyPI metadata is not enough. Preflight focuses on Python version compatibility declared on PyPI for matching releases.

To disable the preflight entirely (e.g. debugging or air‑gapped runs), set the environment variable:

Variable	Effect when set to 1, true, or yes (case-insensitive)
SKIP_PYPI_REQUIRES_PYTHON_CHECK	Skip all PyPI preflight checks; every requirement is passed through to pip wheel.

include_list.yaml

File for additional Python packages to the main requirements list. Built separately to not restrict the main requirements list.

The syntax can be also converted into a sentence: "For assembled main requirements additionally include package_name with version on platform for python version".

build_requirements.txt

File for the requirements needed for the build process and the build script.

os_dependencies

When there is a need for additional OS dependencies to successfully build the wheels on a specific platform and architecture, the .sh script in the os_dependencies directory can be adjusted.

Universal wheel tag - linking of dynamic libraries

The repair tools are used after build to link and bundle all the needed libraries into the wheel to produce correct universal tag and working wheel. If this is not able to achieve the broken wheel is deleted and not published to Espressif's PyPI.

• auditwheel package to repair Linux's manylinux wheels
• delocate package to repair Mac's dynamically linked libraries
• delvewheel package to repair Windows's DLLs

This logic is done by the repair workflow and the repair_wheels.py script

ARMv7 vs ARMv7 Legacy: same wheel filename, different binaries

Linux ARMv7 and Linux ARMv7 Legacy can both produce a wheel whose filename is identical (same PEP 425 tags) while the ELF contents differ (different glibc/OpenSSL/Rust toolchain lineage). Note: wheels-download-directory-* CI artifacts are the pre-repair build outputs; comparing those can still show identical names until the repair workflow runs. Two bad outcomes follow if that is not handled after repair/merge:

1. Artifact merge / local flatten — downloading multiple wheels-repaired-* artifacts into one directory with merge-multiple: true can make the second file silently overwrite the first on disk before any upload runs.
2. S3 upload — upload_wheels.py publishes to pypi/<package>/<wheel-filename>. Uploading a second wheel with the same key replaces the object; clients then see whichever build ran last. ARMv7 vs Legacy clashes must be caught before upload (merge collision check and distinct manylinux_*_armv7l tags when auditwheel allows), not by refusing re-uploads of an existing key when wheel bytes change between CI runs.

Mitigations in this repo:

• Repair sets AUDITWHEEL_PLAT and AUDITWHEEL_ONLY_PLAT per lineage (manylinux_2_36_armv7l vs manylinux_2_31_armv7l) so repair_wheels.py runs auditwheel repair --plat ... --only-plat and emitted wheels get distinct single-tag filenames when auditwheel supports it. If AUDITWHEEL_PLAT is set, ARMv7 “libc detection failed” outcomes are not treated as non-fatal skips (that would leave identical filenames across lineages).
• The repair workflow merges repaired artifacts using per-artifact subdirectories, then runs check_wheel_collisions.py to fail CI only if the same *.whl basename appears with different contents under both wheels-repaired-linux-armv7 and wheels-repaired-linux-armv7legacy and the wheel is not a pure universal (platform tag any only; those ZIPs can legitimately differ between lineages). Other duplicate basenames across macOS runners or pure-Python wheels are expected and are ignored, before flattening for tests/upload.

Activity Diagram

The main file is build-wheels-platforms.yml which is scheduled to run periodically to build Python wheels for any requirement of all ESP-IDF-supported versions.

The diagram was generated with the open-source tool PlantUML (and edited)

Note

Python version dependent wheels explanation

Python dependent wheels are wheels which depend on the CPython’s Application Binary Interface (ABI). These are checked based on the wheel filename format where the abi tag is checked for cp. Such wheels need to be build also for all supported Python versions, not only for the minimum Python version supported by ESP-IDF.

Custom Docker images

Docker files are in its own repository where there are build and published from. https://github.com/espressif/github-esp-dockerfiles

• ARMv7 runner - manylinux_armv7
• ARMv7 runner (older OSes) - manylinux_armv7_legacy
  • For older ARMv7 operating systems
  • For packages requiring glibc 2.31

Warning

piwheels may rely on system-provided shared libraries (i.e. may not bundle .libs/). If a target OS is missing those libraries or has an incompatible version, imports may fail at runtime.