ci: 刷新计时表并把 linux 分片上限抬到 4 —— 3 片已经装不下了 #664
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: validate | |
| on: | |
| pull_request: | |
| # mcpp.toml and index.toml carry the workspace member list, the inherited | |
| # [indices] redirect and the client version floor — a change to any of them | |
| # can break every member, so they gate the run like the descriptors do. | |
| paths: ["pkgs/**/*.lua", "tests/**", "README.md", "README.zh-CN.md", "mcpp.toml", "index.toml", ".github/workflows/validate.yml"] | |
| push: | |
| branches: [main] | |
| schedule: | |
| # nightly full regression — exercises every workspace member regardless of diff | |
| - cron: "0 6 * * *" | |
| workflow_dispatch: | |
| inputs: | |
| cache: | |
| description: "Package build cache — 'local' rebuilds every dependency per member, which is what the timing table should be read against when comparing" | |
| type: choice | |
| options: [global, local] | |
| default: global | |
| env: | |
| # 2026.8.5.4 carries two things this workflow depends on: | |
| # .5.1 `tools = [...]` — how a consumer asks for a dependency's | |
| # `kind = "bin"` target, which tests/examples/protobuf-protoc is built | |
| # on. Before it: "tools must be a string, inline dep table, or nested | |
| # table". | |
| # .5.4 windows links with lld. link.exe caps a response-file LINE at | |
| # 128 KiB and opencv-module / opencv-module-dnn went past it — | |
| # fatal error LNK1170: line in command file contains 135135 or | |
| # more characters | |
| # after 795s / 1166s of compiling. .5.3 newline-separated OUR response | |
| # file, which was necessary but not sufficient: clang, acting as the | |
| # driver, writes a SECOND one for the linker that we do not control. | |
| # lld's response-file parser has no per-line limit at all. | |
| # Together with re-enabling the global package cache below, this is | |
| # what makes a green FULL run possible again: .5.3 removes the | |
| # windows link failure, the cache removes the 150-minute timeout. | |
| # | |
| # Neither of them moves index.toml's min_mcpp: exposing compat.protobuf's `protoc` | |
| # target is additive, and 2026.8.3.3 still parses that descriptor with an | |
| # empty unknown_keys. The floor an index publishes decides whether older | |
| # clients keep working at all (mcpp#349), so it moves only when a descriptor | |
| # genuinely stops being readable — which is not the case here. | |
| # 2026.8.3.1: on macOS, a global object that touches std::cout during static | |
| # init crashes on sight (mcpp#336). Mach-O has no priority-ordered init | |
| # section and libc++'s <iostream> carries no ios_base::Init guard of its own, | |
| # so the streams are still all-zero when an archive member's initializer | |
| # runs. mcpp now links a generated object FIRST whose constructor brings them | |
| # up. It is not fixable package-side — std::ios_base::Init is only | |
| # forward-declared in libc++'s <ios> — so boost-ext.ut genuinely requires | |
| # this floor on macOS, and min_mcpp/latest_mcpp move with the pin as they | |
| # always have (a consumer below the floor would get a segfault with no | |
| # diagnostic, which is worse than E0006). | |
| # 2026.8.3.3 is the pin rather than .3.1: .3.2/.3.3 are cross-compilation | |
| # fixes (PE artifact naming, -static host-vs-target) that no leg of this | |
| # matrix exercises, so taking the newest of the train costs nothing. | |
| # 0.0.109: a bare dependency's wire address takes BOTH halves from the | |
| # descriptor the identity gate accepted (mcpp#286). This is the client-side | |
| # other half of the SPEC-001 migration below: mcpp used to take the NAME from | |
| # the descriptor and the NAMESPACE from the request, so a bare `gtest = | |
| # "1.15.2"` addressed `mcpplibs:gtest` — a key no index has. It only ever | |
| # worked because the pre-migration literal `package.name` read | |
| # "compat.gtest", which the hardcoded `compat.<short>` retry then caught. | |
| # Short names removed that coincidence and left every bare request against | |
| # this index broken on 0.0.108, which is why min_mcpp moves to 0.0.109. | |
| # Note this index cannot cover that spelling itself: the `[indices]` redirect | |
| # is keyed by the REQUEST's namespace, so a bare dependency resolves from the | |
| # published remote index rather than the checkout under test. Bare-name | |
| # resolution against a short-name index is upstream's e2e 165; what moving | |
| # the floor buys here is that consumers of THIS index get a client that can | |
| # address it. | |
| # 0.0.106: SPEC-001 package identity (mcpp#280). `package.name` is a SINGLE | |
| # ATOMIC SEGMENT — all hierarchy lives in `package.namespace` — and mcpp | |
| # addresses a package by the LITERAL name it read, so descriptors no longer | |
| # repeat their namespace inside `name`. This index is migrated to the short | |
| # form, which is why min_mcpp/latest_mcpp move in lock-step: an older client | |
| # re-derives `<ns>.<short>`, misses, and reports a bare E_NOT_FOUND. Bundles | |
| # xlings 0.4.69, which keys its index by (namespace, name) so two packages | |
| # sharing a short name in one index are both addressable (xlings#381) — this | |
| # index now has three such pairs (imgui / ffmpeg / lua under compat vs the | |
| # default namespace). | |
| # 0.0.102: windows command-line ceiling (mcpp#261 — the clang scan rule got | |
| # its P1689 JSON through shell redirection, which forced a `cmd /c` wrapper | |
| # and with it cmd.exe's 8191-char limit; clang-scan-deps -o removes both, and | |
| # $local_includes-carrying rules now fall back to response files). The pin | |
| # matters here because a package consumed FROM the registry sits under a | |
| # ~124-char xpkgs path instead of its own ~23-char checkout, which is what | |
| # pushed the vendored-opencv scan command over the line. Also: purview-include | |
| # depfile tracking extended to Clang (#257 — stale BMI reuse), OS-conditional | |
| # `[build].flags` (#258), and per-OS splices keyed on the resolved target | |
| # rather than the host (#254). | |
| # 0.0.101: per-feature per-glob flags + per-OS features (mcpp#253) — what | |
| # lets opencv select its dnn gemm backend per platform. | |
| # 0.0.99: feature dep/feat forwarding (mcpp#243 — a feature can open a | |
| # feature OF a dependency, e.g. opencv `dnn` forwarding compat.opencv/dnn); | |
| # vendored xlings 0.4.67 for the >=2 index_repo install fix (mcpp#238 / | |
| # openxlings/xlings#374); build.mcpp compiled program named `.exe` on | |
| # Windows (mcpp#230 secondary surface, after the 0.0.96 scanner crash fix). | |
| # 0.0.98: closes the obj-path disambiguation follow-ups that gated the | |
| # source-build compat.opencv unification — #240 (link inputs now follow | |
| # the disambiguated object names, so a dependency + consumer sharing a | |
| # source basename like `src/main.cpp` no longer 'obj/main.o missing') and | |
| # #239 (absolute/`..` dep-generated source paths sanitized component-wise | |
| # so objects stay under obj/). Also: `MCPP_DEP_<NAME>_DIR` build.mcpp | |
| # contract (#241), consumer-side `default-features = false` (#242), and a | |
| # loud unknown-mcpp-key warning with did-you-mean (#237, replaces the | |
| # silent-ignore at build time). Carried from 0.0.97: default-namespace | |
| # index redirect (`[indices] default = { path }`), which turned the public | |
| # module packages (imgui/ffmpeg/opencv/tinyhttps) into ordinary workspace | |
| # members and retired the per-package reseeding smoke shells + their | |
| # dedicated jobs; synchronous nasm bootstrap (mcpp#232 — the `mcpp index | |
| # update` pre-step is gone), obj-path disambiguation (#233), spacey-defines | |
| # quoting (#234), purview-include depfile tracking (#235). Older floors of | |
| # note: 0.0.96 fixed the windows scanner symlink-escape crash (mcpp#230); | |
| # 0.0.94 fixed feature-gated `sources` under `mcpp test` (mcpp#218); 0.0.91 | |
| # added standard = "c++fly" to the resolver grammar, so c++fly descriptors | |
| # get the lint WARN below, not a hard grammar-parse rejection. | |
| # 2026.8.6.1 是本次身份迁移的**前置**,不是顺手升级。 | |
| # | |
| # `Fetcher::install_path(ns, shortName, version)` 的 legacy 扫描此前匹配任何以 | |
| # `-x-<shortName>` 结尾的目录、不看命名空间,于是查 `ocornut:imgui@1.92.8` 会拿到 | |
| # `compat-x-imgui/1.92.8` —— 另一个仅仅短名相同的包。它一直够不到,是因为 module | |
| # 层用打包计数(imgui@0.0.6)而 compat 用上游版本(compat.imgui@1.92.8),版本永远 | |
| # 不撞;本次把 module 层对齐到上游之后它们重合了。mcpp#364 修掉了它。 | |
| # | |
| # index.toml 的 min_mcpp **不动**。两个会撞的 compat 邻居都是 Form B,所以旧客户端 | |
| # 撞上时是响亮报错而不是静默用错包;下限是一道让整个索引对旧客户端失效的闸门 | |
| # (mcpp#349),只该在描述符真的读不动时抬。这里读得动,差的是解析得对。 | |
| # | |
| # 2026.8.8.2 是本 PR 的**前置**,不是顺手升级。 | |
| # | |
| # `compat.glx-runtime` 依赖 mesa,mesa 声明 `xim:glibc@>=2.38`;下限被任何更高 | |
| # 版本满足,于是安装图形栈会在既有 glibc 旁边**再装一个**。2026.8.8.2 之前的 | |
| # mcpp 用 `readdir` 的第一项来解析「那个 glibc payload」,编译侧与产物的 | |
| # interpreter 因此可以指向不同版本 —— 这正是 #179 落地后 `asio-module` 和 | |
| # `core` 变红、并导致整份改动被 #180 revert 的原因。 | |
| # | |
| # 用早于 2026.8.8.2 的 mcpp 重新落地这份改动,就是在仍会犯这个错的引擎上复现 | |
| # 事故条件。tests/check_graphics_install_side_effects.sh 也需要它 —— 该测试在 | |
| # 更旧的 mcpp 上失败是**有意的**。 | |
| MCPP_VERSION: "2026.8.8.2" | |
| jobs: | |
| lint: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Install lua | |
| run: sudo apt-get install -y --no-install-recommends lua5.4 | |
| - name: Lint package descriptors | |
| run: | | |
| fail=0 | |
| for f in pkgs/*/*.lua; do | |
| # 1. Lua syntax check — load (= compile) without executing. | |
| # `loadfile(name, 't')` rejects bytecode and parses text only. | |
| if ! lua5.4 -e "assert(loadfile('$f', 't'))" >/dev/null 2>&1; then | |
| echo "::error file=$f::lua syntax error" | |
| fail=1 | |
| fi | |
| # 2. xpkg V1 baseline: the file has to populate `package = { ... }` | |
| # with at least `spec`, `name`, and an `xpm` table. Form A vs | |
| # Form B (mcpp = "<path>" / mcpp = { ... }) is descriptor-author | |
| # choice and not enforced here. | |
| for needle in 'spec *=' 'name *=' 'xpm *='; do | |
| if ! grep -q "$needle" "$f"; then | |
| echo "::error file=$f::missing required field ($needle)" | |
| fail=1 | |
| fi | |
| done | |
| # 3. Package version identifiers and dependency versions should be | |
| # bare versions ("1.2.3"), not upstream tag names ("v1.2.3"). | |
| # Download URLs may still contain refs/tags/v* when upstream | |
| # uses that tag spelling. | |
| if grep -nE '\["v[0-9]+|\["[^"]+"\][[:space:]]*=[[:space:]]*"v[0-9]+' "$f"; then | |
| echo "::error file=$f::version identifiers must not use a leading v" | |
| fail=1 | |
| fi | |
| # 4. Mirror table sanity: when a download `url` is written as a | |
| # { GLOBAL=..., CN=... } table, both regions must be present and | |
| # the CN entry must point at the gitcode mcpp-res mirror. | |
| if ! lua5.4 tests/check_mirror_urls.lua "$f"; then | |
| fail=1 | |
| fi | |
| # 5. `name` must be a SINGLE ATOMIC SEGMENT; hierarchy belongs in | |
| # `namespace` (mcpp SPEC-001 §3.2). The legacy fully-qualified | |
| # spelling stays accepted. Cheap second gate: it runs before the | |
| # pinned mcpp is even downloaded, and mcpp >= 0.0.106 enforces | |
| # the same rule inside `mcpp xpkg parse`. | |
| if ! lua5.4 tests/check_package_name.lua "$f"; then | |
| fail=1 | |
| fi | |
| # 6. c++fly admission policy (mcpp design 2026-07-14 §11-Q2, v1): | |
| # c++fly means "toolchain's latest level + every experimental | |
| # gate" — deliberately toolchain-dependent, so a published | |
| # package built with it is not reproducible for consumers. | |
| # Policy: WARN (never fail) and observe ecosystem usage before | |
| # deciding whether to tighten. Two spellings: `language = ` is | |
| # the descriptor's inline mcpp-segment key; `standard = ` covers | |
| # mcpp.toml content embedded in heredoc/generated_files blocks. | |
| if grep -nE '\b(language|standard)[[:space:]]*=[[:space:]]*"c\+\+fly"' "$f" >/dev/null; then | |
| echo "::warning file=$f::declares C++ standard \"c++fly\" (experimental playground mode) — toolchain-dependent and non-reproducible for consumers; published packages should pin a concrete standard (c++23/c++26)" | |
| fi | |
| done | |
| [ $fail -eq 0 ] && echo "All package files valid." | |
| exit $fail | |
| # ── Whole-repository check (needs every descriptor at once) ────── | |
| # An install() hook addressing a sibling package does so by | |
| # `<namespace>:<literal package.name>`, and a miss returns nil rather | |
| # than raising — so a stale spelling surfaces far downstream (a broken | |
| # libxcb showed up as a libX11 link error). Verified across the repo | |
| # because it needs the full set of declared identities. | |
| - name: Lint cross-package references | |
| run: lua5.4 tests/check_cross_package_refs.lua pkgs/*/*.lua | |
| # ── Partial version bumps ──────────────────────────────────────── | |
| # A bump is a one-line-looking edit that has to land in N platform | |
| # sections. Editing `xpm.linux` and reading the file back gives a file | |
| # that CONTAINS the new version, so the author -- and any whole-file | |
| # grep -- sees success, while the other platforms are left behind. The | |
| # failure then surfaces as `<pkg>@<ver> not found` on a platform, | |
| # against a file that literally contains that version string. | |
| # | |
| # Measured 2026-08-06: xpkg 0.0.52 and 0.0.53 were both added to | |
| # `xpm.linux` alone; linux CI went green twice while macOS and Windows | |
| # failed, and eight checks made from the outside all came back correct | |
| # because each asked "is it in the index?" instead of "is it in THIS | |
| # platform's section?". Whole-repo, because a partial bump is only | |
| # visible by comparing sections against each other. | |
| - name: Lint platform version parity | |
| run: lua5.4 tests/check_platform_version_parity.lua pkgs/*/*.lua | |
| # ── Single-source-of-truth grammar check ───────────────────────── | |
| # `mcpp xpkg parse` uses EXACTLY the resolver's parser, so what | |
| # passes here is what builds for users of the pinned MCPP_VERSION. | |
| # Strict by default: unknown mcpp-segment keys fail (they would be | |
| # silently ignored at build time). This also mechanically enforces | |
| # the rollout rule "floor first, new grammar after": descriptors | |
| # needing a newer grammar cannot pass a lint pinned to an older mcpp. | |
| - name: Download pinned mcpp | |
| run: | | |
| curl -L -fsS -o mcpp.tar.gz \ | |
| "https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/mcpp-${MCPP_VERSION}-linux-x86_64.tar.gz" | |
| tar -xzf mcpp.tar.gz | |
| echo "MCPP=$PWD/mcpp-${MCPP_VERSION}-linux-x86_64/bin/mcpp" >> "$GITHUB_ENV" | |
| - name: Parse descriptors with the resolver grammar (mcpp xpkg parse) | |
| run: | | |
| fail=0 | |
| for f in pkgs/*/*.lua; do | |
| if ! "$MCPP" xpkg parse "$f" > /dev/null; then | |
| echo "::error file=$f::mcpp xpkg parse failed (resolver grammar)" | |
| fail=1 | |
| fi | |
| done | |
| [ $fail -eq 0 ] && echo "All descriptors parse with mcpp ${MCPP_VERSION}." | |
| exit $fail | |
| mirror-cn-reachable: | |
| # Closed-loop guard for the CN mirror: every CN url referenced by a | |
| # descriptor must be a live, downloadable gitcode release asset. | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Install lua | |
| run: sudo apt-get install -y --no-install-recommends lua5.4 | |
| - name: Check CN mirror assets are reachable | |
| run: | | |
| fail=0 | |
| # collect unique CN urls across all descriptors | |
| : > /tmp/cn.tsv | |
| for f in pkgs/*/*.lua; do | |
| lua5.4 tests/list_cn_urls.lua "$f" >> /tmp/cn.tsv || true | |
| done | |
| sort -u /tmp/cn.tsv -o /tmp/cn.tsv | |
| total=$(grep -c . /tmp/cn.tsv || true) | |
| echo "checking $total CN mirror url(s)" | |
| while IFS=$'\t' read -r url sha; do | |
| [ -z "$url" ] && continue | |
| # follow redirects; gitcode release assets resolve to object storage | |
| code=$(curl -fsSL -o /dev/null -w '%{http_code}' --retry 2 --max-time 60 "$url" || echo "000") | |
| if [ "$code" != "200" ]; then | |
| echo "::error::CN mirror unreachable ($code): $url" | |
| fail=1 | |
| else | |
| echo "ok: $url" | |
| fi | |
| done < /tmp/cn.tsv | |
| [ $fail -eq 0 ] && echo "All CN mirror urls reachable." | |
| exit $fail | |
| # ── The whole test surface, as a mcpp workspace ─────────────────────── | |
| # mcpp-index is a mcpp [workspace]; every per-library test project under | |
| # tests/examples/ is a member. `mcpp test --workspace` builds + runs each | |
| # member's tests/ (behavioral assertions) on each OS — members self-gate by | |
| # `[target.'cfg(...)']` (e.g. the X11/glfw stack is linux-only, openblas is | |
| # windows-only), so one command covers the matrix with no shell driver. | |
| # The ~/.mcpp/registry cache carries the built compat packages (xpkgs) across | |
| # runs, so repeat builds are fast. | |
| # | |
| # timeout-minutes is sized for the COLD build, not the cached path. The opencv | |
| # module package carries a from-source OpenCV 5 build, and each feature variant | |
| # re-keys the store into a full recompile, so a full run (forced whenever this | |
| # workflow file changes — e.g. a version bump) serially builds three OpenCV | |
| # variants on one runner: the opencv-module base member plus the `unifont` and | |
| # `dnn` feature members. The registry cache (restore-keys prefix below) | |
| # amortizes those across subsequent runs. 150 covers the one-time cold full | |
| # build with headroom; it is a ceiling, not a target. | |
| # ── The plan, computed ONCE ─────────────────────────────────────────── | |
| # Was inlined in every workspace job — three runners each re-deriving the | |
| # same answer. It now also has to be decided BEFORE the matrix exists, | |
| # because the matrix's shard dimension depends on it: a full run fans out, | |
| # a selective one does not. | |
| select: | |
| runs-on: ubuntu-latest | |
| outputs: | |
| members: ${{ steps.fanout.outputs.members }} | |
| matrix: ${{ steps.fanout.outputs.matrix }} | |
| plan: ${{ steps.plan_shards.outputs.plan }} | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: Install lua | |
| run: sudo apt-get install -y --no-install-recommends lua5.4 | |
| # ── Selective member testing ────────────────────────────────────── | |
| # `mcpp test --workspace` builds every member (opencv, ffmpeg, …) and | |
| # dominates CI wall-clock, while a PR almost always touches one | |
| # package. Map changed files → affected members and test only those: | |
| # pkgs/<x>/<lib>.lua → members whose mcpp.toml references <lib> | |
| # tests/examples/<m>/** → member <m> | |
| # Run the FULL workspace when the change can affect everything: | |
| # non-PR events (push to main, the nightly cron, dispatch), this | |
| # workflow file (it carries the mcpp version pins, so a version bump | |
| # always re-validates every package), a non-member edit to the | |
| # workspace manifest, or shared test scripts. Docs-only and tools/-only | |
| # changes select nothing. | |
| # Note: bash 3.2 on macOS runners — no associative arrays here. | |
| - name: Select affected workspace members | |
| id: plan | |
| shell: bash | |
| run: | | |
| full() { echo "MEMBERS=__ALL__" >> "$GITHUB_ENV"; echo "full run: $1"; exit 0; } | |
| [ "${{ github.event_name }}" = "pull_request" ] || full "event=${{ github.event_name }}" | |
| base="origin/${{ github.base_ref }}" | |
| changed=$(git diff --name-only "$base"...HEAD) | |
| printf 'changed files vs %s:\n%s\n' "$base" "$changed" | |
| sel="" | |
| add() { case " $sel " in *" $1 "*) ;; *) sel="$sel $1" ;; esac; } | |
| while IFS= read -r f; do | |
| [ -n "$f" ] || continue | |
| case "$f" in | |
| .github/workflows/validate.yml|tests/*.sh) full "$f" ;; | |
| mcpp.toml) | |
| # Workspace manifest. Every new-package PR appends to the | |
| # members list, so that alone must NOT force a full run: | |
| # select the added members; anything else in this file | |
| # (indices, settings) affects everyone → full. | |
| if ! diff -q <(git show "$base:mcpp.toml" | grep -v 'tests/examples/') \ | |
| <(grep -v 'tests/examples/' mcpp.toml) >/dev/null; then | |
| full "mcpp.toml non-member change" | |
| fi | |
| for p in $(comm -13 <(git show "$base:mcpp.toml" | grep -o 'tests/examples/[A-Za-z0-9._-]*' | sort -u) \ | |
| <(grep -o 'tests/examples/[A-Za-z0-9._-]*' mcpp.toml | sort -u)); do | |
| add "${p#tests/examples/}" | |
| done ;; | |
| tests/examples/*) | |
| m=${f#tests/examples/}; m=${m%%/*} | |
| # A deleted/renamed member dir implies a mcpp.toml edit, | |
| # which already forces a full run above. | |
| [ -d "tests/examples/$m" ] && add "$m" ;; | |
| pkgs/*.lua|pkgs/*/*.lua) | |
| lib=$(basename "$f" .lua); lib=${lib#compat.} | |
| hit=0 | |
| for mt in tests/examples/*/mcpp.toml; do | |
| if grep -q "$lib" "$mt"; then add "$(basename "$(dirname "$mt")")"; hit=1; fi | |
| done | |
| [ "$hit" = 1 ] || echo "note: no workspace member exercises $f" ;; | |
| # tools/ holds OFFLINE descriptor-generation and publishing | |
| # helpers (tools/compat-*/, tools/gtc/, publish_mcpp_index.sh). | |
| # Nothing under it is consumed by a package build: when one of | |
| # them actually changes a package, the generated pkgs/*.lua | |
| # changes with it and the rule above selects the right members. | |
| # So a tools/ edit alone selects nothing rather than forcing a | |
| # full workspace rebuild. | |
| # The timing table decides how work is DISTRIBUTED across | |
| # shards, never what gets built — no member's result can change | |
| # because a measured number moved. It used to fall through to | |
| # the `*)` catch-all below and force a full run, which is the | |
| # most expensive way in this workflow to test nothing: the next | |
| # full run reads the new numbers anyway. | |
| # | |
| # Note this makes the file invisible to CI. A malformed row is | |
| # then silent, and `plan_shards` prices anything it cannot parse | |
| # at the median — the exact failure mode that overflowed a shard | |
| # here. Guard it in `lint` if that ever bites. | |
| tests/member-timings.tsv) : ;; | |
| *.md|docs/*|.agents/*|.github/*|tools/*) : ;; | |
| *) full "unclassified change: $f" ;; | |
| esac | |
| done <<EOF | |
| $changed | |
| EOF | |
| sel=${sel# } | |
| echo "MEMBERS=$sel" >> "$GITHUB_ENV" | |
| echo "selected members: ${sel:-<none>}" | |
| # Sharding is for the FULL run only, and the shard count per platform is | |
| # that platform's RUNNER CONCURRENCY — not a round number. | |
| # | |
| # Measured on this repo (24 jobs queued, 6 running): | |
| # macos 1 · linux 3 · windows 2 | |
| # | |
| # That measurement is what makes over-sharding a real cost rather than a | |
| # theoretical one: at concurrency 1, eight macOS shards run BACK TO BACK | |
| # and each pays its own checkout + mcpp download + cache restore, so the | |
| # split is strictly slower than not splitting. Wall-clock is | |
| # ceil(shards / concurrency) x slowest-shard; shards beyond the | |
| # concurrency only add fixed cost. | |
| # | |
| # Re-measure with: | |
| # gh api repos/<owner>/<repo>/actions/runs/<id>/jobs --paginate \ | |
| # --jq '[.jobs[]|select(.status=="in_progress")]|length' | |
| - name: Decide the fan-out | |
| id: fanout | |
| shell: bash | |
| run: | | |
| full=0; [ "$MEMBERS" = "__ALL__" ] && full=1 | |
| emit() { # platform os suffix ext mcpp xlings toolchain shards | |
| for i in $(seq 0 $(( $8 - 1 ))); do | |
| printf '{"platform":"%s","os":"%s","suffix":"%s","ext":"%s","mcpp":"%s","xlings":"%s","toolchain":"%s","shard":%d,"shards":%d},' \ | |
| "$1" "$2" "$3" "$4" "$5" "$6" "$7" "$i" "$8" | |
| done | |
| } | |
| # Shard count follows the WORK, per platform, measured. | |
| # | |
| # It used to be binary — full run: linux 3 / macos 1 / windows 2, | |
| # anything else: 1 each — which asserts two things that are not | |
| # true. "A partial run is small": touching a widely consumed | |
| # descriptor selects every member that consumes it, and this PR's | |
| # `pkgs/c/compat.openssl.lua` selected four; one linux shard took | |
| # grpc-codegen (3563s) and grpc-module (1701s) back to back and was | |
| # cancelled at exactly 1h30m, the job cap. "A full macOS run fits in | |
| # one shard": at 6922s measured it does not — it fit only while the | |
| # total sat just under the cap, and the run that added grpc-codegen | |
| # (1715s on macOS) pushed it to 1h30m20s. | |
| # | |
| # Sharding is not only a wall-clock lever. Each shard is its own job | |
| # with its own `timeout-minutes`, so it is also how the work is made | |
| # to FIT. That is the half the "macOS concurrency is 1, so extra | |
| # shards run back to back" note above left out: back to back is fine | |
| # when the alternative is not finishing. | |
| # | |
| # tests/member-timings.tsv already holds every member's measured | |
| # cost and plan_shards.lua already packs by it (LPT), so the fan-out | |
| # comes from the same table: sum this run's members for that | |
| # platform and take one shard per ~70 minutes, which leaves ~20 | |
| # minutes of headroom under the 90-minute cap for a cold cache. | |
| # | |
| # The caps are where runner concurrency comes back in: linux 4, | |
| # macOS 2, windows 2. Past those, more shards buy fit that is | |
| # already there and pay another checkout + mcpp download + cache | |
| # restore. | |
| # | |
| # linux was 3 — "the measured concurrency, a 4th shard would queue". | |
| # That cap outlived its premise twice over: the workspace grew, and | |
| # the second toolchain leg means linux emits 2 x N jobs, so it | |
| # queues at any N. It is now 4, which is what the formula asks for. | |
| # | |
| # Full run, from the table: linux 15891s over 4 -> ~74min slowest | |
| # shard, macOS 6922s over 2 -> ~57, windows 8043s over 2 -> ~67. | |
| # | |
| # linux is still the tight one, and the number to watch is the | |
| # SLOWEST shard against the 90-minute job cap, not the average. | |
| # Simulated on the current table: 3-way 98min, 4-way 74, 5-way 60. | |
| # The levers, in order: raise the linux cap to 5, then the job | |
| # timeout. | |
| # | |
| # Keep tests/member-timings.tsv fresh — that is not housekeeping. An | |
| # untimed member is priced at the MEDIAN, so a heavy newcomer packs | |
| # like a trivial one. `mysql-connector-cpp` (881s) and | |
| # `libmysqlclient` (329s) landed on shard 0 next to grpc-codegen | |
| # exactly that way and pushed it past the cap, and the stale table | |
| # then hid the overflow by under-counting the total. | |
| # `plan_shards.lua <platform> 0 0` reports the platform's total cost. | |
| # | |
| # This used to sum the timing table's rows inline, which was the same | |
| # answer only while the table listed every member. It now lists just | |
| # the heavy ones, so summing rows measures a fraction of the work: | |
| # 10965s instead of 16095s on linux, i.e. 3 shards where 4 are | |
| # needed, i.e. a slowest shard of 107 minutes against a 90-minute | |
| # cap. Asking plan_shards keeps ONE definition of both "which members | |
| # are there" and "what does an untimed one cost". | |
| # | |
| # The estimate is tuned on linux (16095s vs 15891s measured, +1.3%). | |
| # macOS and windows members are cheaper on average than the default, | |
| # so their totals come out high — harmless here because both are | |
| # already at their cap, and erring toward more shards is the safe | |
| # direction. Revisit if either cap moves. | |
| shards_for() { # platform cap -> shard count | |
| local secs sel="" | |
| [ "$MEMBERS" = "__ALL__" ] || sel="$MEMBERS" | |
| secs=$(lua5.4 tests/plan_shards.lua "$1" 0 0 $sel) | |
| local n=$(( secs / 4200 + 1 )) | |
| [ "$n" -gt "$2" ] && n="$2" | |
| echo "$1 work: ${secs}s (measured) -> $n shard(s)" >&2 | |
| echo "$n" | |
| } | |
| # linux 3 -> 4. The cap was the binding constraint, not the ~70min | |
| # target: `secs / 4200 + 1` asks for 4 on the refreshed table | |
| # (15891s) and has been getting clamped back to 3. Simulated on that | |
| # table, slowest shard: 3-way 98min (OVER the 90min job cap — and | |
| # `linux default 0/3` was in fact cancelled at 90min on run | |
| # 31266814148), 4-way 74min, 5-way 60min. 4 is the formula's own | |
| # answer and leaves 16 minutes for a cold cache; 5 is there if that | |
| # stops being enough. | |
| # | |
| # This does cross the concurrency line the old comment drew: with two | |
| # toolchain legs linux now emits 8 jobs against a measured runner | |
| # concurrency of 3, so shards queue. Queuing is the right trade — | |
| # back-to-back shards still finish, a shard over the cap does not. | |
| ln=$(shards_for linux 4) | |
| mn=$(shards_for macos 2) | |
| wn=$(shards_for windows 2) | |
| { | |
| printf '{"include":[' | |
| # linux runs TWICE: once on mcpp's default toolchain and once on | |
| # llvm. Not redundancy — the two differ in a way that decides | |
| # whether a descriptor is correct. mcpp's linux default (gcc) | |
| # reaches the compiler through --sysroot into a clean xlings | |
| # subos, so the HOST /usr/include is not on the search path; | |
| # llvm has no sysroot and the host's headers are. compat.ffmpeg | |
| # put its vendored source root on -idirafter (below the system | |
| # dirs) and was green here for months while being broken on any | |
| # developer box with libavutil-dev installed — the gcc leg | |
| # structurally cannot see that class of bug. | |
| # | |
| # Cost, measured: this doubles linux to 6 jobs (2 x 3 shards) at | |
| # ~75min each, against a measured linux runner concurrency of 3 — | |
| # so the second leg QUEUES behind the first rather than running | |
| # beside it, and a full run's linux wall-clock roughly doubles. | |
| # That is the price of the coverage; the levers if it hurts are | |
| # in the shards_for comment above. | |
| emit linux ubuntu-latest linux-x86_64 tar.gz bin/mcpp registry/bin/xlings default "$ln" | |
| emit linux ubuntu-latest linux-x86_64 tar.gz bin/mcpp registry/bin/xlings llvm "$ln" | |
| emit macos macos-15 macosx-arm64 tar.gz bin/mcpp registry/bin/xlings default "$mn" | |
| emit windows windows-latest windows-x86_64 zip bin/mcpp.exe registry/bin/xlings.exe default "$wn" | |
| printf ']}' | |
| } | sed 's/,]}/]}/' > /tmp/matrix.json | |
| echo "matrix=$(cat /tmp/matrix.json)" >> "$GITHUB_OUTPUT" | |
| echo "members=$MEMBERS" >> "$GITHUB_OUTPUT" | |
| cat /tmp/matrix.json | |
| # The split is computed ONCE, here, and shipped to the runners as data. | |
| # It used to run on each runner, which needed lua5.4 on all three | |
| # platforms — windows has no apt or brew, and macOS's brew installs | |
| # `lua`, not `lua5.4`, so every non-linux shard died with | |
| # `lua5.4: command not found` after 16 seconds. Deciding once is also | |
| # simply correct: one plan, not three runners each re-deriving it. | |
| - name: Plan the shards | |
| id: plan_shards | |
| shell: bash | |
| run: | | |
| plan='${{ steps.fanout.outputs.members }}' | |
| [ "$plan" = "__ALL__" ] && plan="" | |
| { | |
| printf '{' | |
| first=1 | |
| # `.shards`, NOT the number of matrix entries. | |
| # | |
| # These stopped being the same number the moment linux grew a | |
| # second toolchain leg: the platform emits 2 x 3 = 6 entries while | |
| # the split is still 3 ways, and every entry carries shard 0..2. | |
| # Planning 6 ways and consuming three of them dropped 36 of 65 | |
| # members on the floor — silently, because a member that is never | |
| # assigned is indistinguishable from one that passed. `ffmpeg`, | |
| # `opencv-module*`, `catch2-v2` and `openssl` were among them, on | |
| # the very run that added the leg meant to test them. | |
| # | |
| # `.shards` is the value emit() already wrote per entry and the | |
| # same one the job reads for its own `matrix.shards`, so plan and | |
| # consumer now read one number instead of two that agreed by | |
| # accident. | |
| for spec in linux:$(jq -r 'first(.include[]|select(.platform=="linux")).shards' /tmp/matrix.json) \ | |
| macos:$(jq -r 'first(.include[]|select(.platform=="macos")).shards' /tmp/matrix.json) \ | |
| windows:$(jq -r 'first(.include[]|select(.platform=="windows")).shards' /tmp/matrix.json); do | |
| p=${spec%%:*}; n=${spec##*:} | |
| [ "$first" = 1 ] || printf ',' | |
| first=0 | |
| printf '"%s":{' "$p" | |
| for i in $(seq 0 $((n - 1))); do | |
| [ "$i" = 0 ] || printf ',' | |
| m=$(lua5.4 tests/plan_shards.lua "$p" "$i" "$n" $plan) | |
| printf '"%s":"%s"' "$i" "$m" | |
| done | |
| printf '}' | |
| done | |
| printf '}' | |
| } > /tmp/plan.json | |
| echo "plan=$(cat /tmp/plan.json)" >> "$GITHUB_OUTPUT" | |
| jq . /tmp/plan.json | |
| # Installing the graphics stack must not change what UNRELATED members link | |
| # against. This is the check #179 did not have: every test in this repo asks | |
| # about its own package, so nothing asked whether installing one changes | |
| # things for everyone else -- and `asio-module` / `core` went red on a | |
| # change that never touched them. | |
| # | |
| # Its own job, on Linux, because it needs a real toolchain and a home it can | |
| # watch payloads appear in. It reports INCONCLUSIVE (and fails) rather than | |
| # green whenever its preconditions do not hold; see the script. | |
| graphics-side-effects: | |
| name: "graphics install: no side effects on unrelated members" | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 60 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Download mcpp | |
| shell: bash | |
| env: | |
| MCPP_ARCHIVE: mcpp-${{ env.MCPP_VERSION }}-linux-x86_64.tar.gz | |
| MCPP_ROOT: mcpp-${{ env.MCPP_VERSION }}-linux-x86_64 | |
| run: | | |
| curl -L -fsS -o "$MCPP_ARCHIVE" \ | |
| "https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/${MCPP_ARCHIVE}" | |
| tar -xzf "$MCPP_ARCHIVE" | |
| root="$PWD/$MCPP_ROOT" | |
| # MCPP_HOME is the tarball root, not ~/.mcpp. | |
| # | |
| # A released mcpp is self-contained: it resolves its registry from | |
| # beside its own executable. Copying the payload tree into ~/.mcpp | |
| # and pointing MCPP_HOME there therefore watches one home while the | |
| # build uses another -- which this check detects and refuses to call | |
| # green (it did, on the first run). Aligning them is the fix; the | |
| # check was right. | |
| echo "MCPP=$root/bin/mcpp" >> "$GITHUB_ENV" | |
| echo "MCPP_HOME=$root" >> "$GITHUB_ENV" | |
| echo "MCPP_VENDORED_XLINGS=$root/registry/bin/xlings" >> "$GITHUB_ENV" | |
| - name: Side-effect check | |
| shell: bash | |
| run: | | |
| "$MCPP" self config --mirror GLOBAL 2>/dev/null || true | |
| bash tests/check_graphics_install_side_effects.sh | |
| workspace: | |
| # The shard suffix appears only when the platform is actually split, and | |
| # the toolchain only when it is not mcpp's default — so the existing job | |
| # names are unchanged and only the new llvm leg is labelled. | |
| name: workspace (${{ matrix.platform }}${{ matrix.toolchain == 'default' && '' || format(' {0}', matrix.toolchain) }}${{ matrix.shards == 1 && '' || format(' {0}/{1}', matrix.shard, matrix.shards) }}) | |
| needs: select | |
| if: needs.select.outputs.members != '' | |
| runs-on: ${{ matrix.os }} | |
| # One shard is a fraction of the work, so this is a real ceiling rather | |
| # than the thing that decides whether the job finishes (a full linux run | |
| # used to hit 150 exactly and get cancelled). | |
| timeout-minutes: 90 | |
| strategy: | |
| fail-fast: false | |
| # Whole matrix from `select`: the shard count is per-platform, because it | |
| # tracks that platform's runner concurrency. | |
| matrix: ${{ fromJSON(needs.select.outputs.matrix) }} | |
| steps: | |
| # Full history: the member-selection step below diffs against the PR | |
| # base to decide which workspace members to test. | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| # The cache key is computed ONCE, here, instead of inline in the cache | |
| # step. `hashFiles()` globs the WORKING TREE, and actions/cache | |
| # re-evaluates its `key` in the post (save) step — i.e. AFTER the build, | |
| # when `tests/**` no longer matches 80-odd tracked sources but tens of | |
| # thousands of build-output files under tests/examples/*/target and | |
| # .mcpp (multi-GB; .gitignore does not apply to hashFiles). Hashing that | |
| # tree blew past the runner's 120s template-evaluation cap on windows | |
| # and failed an otherwise all-green job: | |
| # "hashFiles('pkgs/**/*.lua, tests/**, .github/workflows/validate.yml') | |
| # couldn't finish within 120 seconds" | |
| # `git ls-files -s` reads the INDEX, so it sees exactly the tracked | |
| # inputs, never build output, and reports blob SHAs git already has — | |
| # no file content is read at all. Freezing the result in the job env | |
| # also guarantees the save step keys on the same string the restore | |
| # step used, no matter what the build left behind. | |
| - name: Compute registry cache key | |
| shell: bash | |
| run: | | |
| # git hash-object rather than sha256sum/cut: git is already a hard | |
| # requirement here (checkout ran), coreutils on the windows leg is | |
| # only a Git-Bash convenience. | |
| h=$(git ls-files -s -- 'pkgs/**/*.lua' 'tests/**' '.github/workflows/validate.yml' \ | |
| | git hash-object --stdin) | |
| # matrix.toolchain is in the key: both linux legs are runner.os | |
| # Linux, and the cache holds TOOLCHAINS and built compat packages. | |
| # Sharing one entry would let the gcc leg's objects answer for the | |
| # llvm leg — exactly the confusion this second leg exists to catch. | |
| echo "REGISTRY_CACHE_KEY=mcpp-registry-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}-$h" >> "$GITHUB_ENV" | |
| - name: Restore mcpp registry cache | |
| uses: actions/cache@v4 | |
| with: | |
| # Holds toolchains AND the built compat packages (data/xpkgs), so a | |
| # repeat `mcpp test` rebuilds little. | |
| path: ~/.mcpp/registry | |
| key: ${{ env.REGISTRY_CACHE_KEY }} | |
| restore-keys: | | |
| mcpp-registry-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}- | |
| # Host tools built from source (protoc, grpc_cpp_plugin, …). | |
| # | |
| # Measured on the run that added grpc-codegen: 636s to build protoc and | |
| # 660s to build grpc_cpp_plugin — 1296s of a 3363s member, repeated in | |
| # every job of every run, because the registry cache above covers | |
| # ~/.mcpp/registry and the tool store lives in ~/.mcpp/build-cache. | |
| # `protobuf-protoc` and `grpc-module` build the SAME protoc, so a full | |
| # run pays for it several times over. | |
| # | |
| # A coarse rolling key is safe here, which is why this is cheap. The | |
| # store is content-addressed by `<pkg>@<version>/<hash>` and mcpp | |
| # validates an entry FIELD BY FIELD against the recorded entry.json | |
| # (epoch, target, host triple, compiler identity, profile, features, | |
| # transitive dependency closure). A stale entry is not used; it is | |
| # rebuilt. The worst case of a cache miss-match is the status quo. | |
| # | |
| # Deliberately NOT build-cache/v1/pkg: 6.0 GB measured locally against | |
| # Actions' 10 GB per-repo budget, which would evict the registry cache | |
| # this job needs more. The tool store is 116 MB. | |
| - name: Cache host tool store | |
| uses: actions/cache@v4 | |
| with: | |
| path: ~/.mcpp/build-cache/v1/tool | |
| key: mcpp-toolstore-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}-${{ github.run_id }}-${{ matrix.platform }}-${{ matrix.shard }} | |
| restore-keys: | | |
| mcpp-toolstore-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}- | |
| - name: Download mcpp | |
| shell: bash | |
| env: | |
| MCPP_ARCHIVE: mcpp-${{ env.MCPP_VERSION }}-${{ matrix.suffix }}.${{ matrix.ext }} | |
| MCPP_ROOT: mcpp-${{ env.MCPP_VERSION }}-${{ matrix.suffix }} | |
| run: | | |
| curl -L -fsS -o "$MCPP_ARCHIVE" \ | |
| "https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/${MCPP_ARCHIVE}" | |
| case "$MCPP_ARCHIVE" in | |
| *.zip) powershell -NoProfile -Command "Expand-Archive -Force -Path '${MCPP_ARCHIVE}' -DestinationPath '.'" ;; | |
| *) tar -xzf "$MCPP_ARCHIVE" ;; | |
| esac | |
| root="$PWD/$MCPP_ROOT" | |
| mkdir -p "$HOME/.mcpp/registry" | |
| cp -a "$root/registry/." "$HOME/.mcpp/registry/" | |
| # Point mcpp AT the home the caches above cover. | |
| # | |
| # Without this the copy above is one-way scenery: a released mcpp is | |
| # self-contained and resolves its registry from beside its own | |
| # executable, so every build used <tarball>/registry while the cache | |
| # restored, and saved, ~/.mcpp/registry. Proof from the run that | |
| # added grpc-codegen: abseil compiled from | |
| # `<tarball>/registry/data/xpkgs/compat-x-abseil/...`, and | |
| # xim:glibc@2.44 / xim:python@3.13.12 were downloaded again on a job | |
| # that reported a registry cache HIT. | |
| # | |
| # With MCPP_HOME set, the restored registry is the one in play — so | |
| # payloads stop being re-downloaded per job, and the tool store cache | |
| # below lands where the build looks for it. | |
| echo "MCPP_HOME=$HOME/.mcpp" >> "$GITHUB_ENV" | |
| if [[ "$RUNNER_OS" == "Windows" ]]; then | |
| echo "MCPP=$(cygpath -m "$root/${{ matrix.mcpp }}")" >> "$GITHUB_ENV" | |
| echo "MCPP_VENDORED_XLINGS=$(cygpath -m "$root/${{ matrix.xlings }}")" >> "$GITHUB_ENV" | |
| echo "$(cygpath -m "$root/bin")" >> "$GITHUB_PATH" | |
| else | |
| echo "MCPP=$root/${{ matrix.mcpp }}" >> "$GITHUB_ENV" | |
| echo "MCPP_VENDORED_XLINGS=$root/${{ matrix.xlings }}" >> "$GITHUB_ENV" | |
| echo "$root/bin" >> "$GITHUB_PATH" | |
| fi | |
| # compat.ffmpeg / compat.opencv5 carry NASM .asm sources. No host | |
| # install and no index-refresh pre-step needed: mcpp >= 0.0.97 | |
| # resolves nasm itself through the same synchronous gate as the | |
| # toolchain (index refresh + install + payload check BEFORE the build | |
| # plans, mcpp#232). The sandbox copy lands in ~/.mcpp/registry, so | |
| # the cache carries it across runs. | |
| # ── Make the llvm leg look like a developer's box ───────────────── | |
| # A bare GitHub runner has no libav* under /usr/include, so a | |
| # descriptor that loses its vendored headers to the host's would still | |
| # be green here. That is precisely how compat.ffmpeg's -idirafter hole | |
| # survived: nothing in CI had a system ffmpeg to lose to. Installing | |
| # the dev headers is what turns this leg into a real test of include | |
| # ORDER rather than of include EXISTENCE. | |
| # | |
| # Deliberately only on the llvm leg. The default leg reaches its | |
| # compiler through --sysroot, so it is immune by construction and | |
| # installing there would change an existing job's environment for no | |
| # coverage. Keeping the blast radius to the new leg means a red here | |
| # is always about the thing this leg was added to watch. | |
| - name: Install host ffmpeg dev headers (llvm leg only) | |
| if: matrix.platform == 'linux' && matrix.toolchain == 'llvm' | |
| shell: bash | |
| run: | | |
| sudo apt-get update -qq | |
| sudo apt-get install -y -qq \ | |
| libavutil-dev libavcodec-dev libavformat-dev \ | |
| libavfilter-dev libavdevice-dev libswscale-dev libswresample-dev | |
| dpkg -l | grep -E 'libav(util|codec)-dev|libswscale-dev' | |
| # ── Select the toolchain for this leg ───────────────────────────── | |
| # No version pin: `llvm` resolves to whatever the toolchain family's | |
| # current release is for the PINNED MCPP_VERSION, which is the version | |
| # that mcpp itself vouches for. Pinning a concrete llvm here would fail | |
| # hard the day that mcpp release stops shipping it, and the point of | |
| # this leg is the ABSENCE of a sysroot, not a particular llvm. | |
| - name: Select toolchain | |
| if: matrix.toolchain != 'default' | |
| shell: bash | |
| run: | | |
| "$MCPP" toolchain install ${{ matrix.toolchain }} | |
| "$MCPP" toolchain default ${{ matrix.toolchain }} | |
| "$MCPP" toolchain list | |
| # ── This shard's slice of the plan ──────────────────────────────── | |
| # `select` decided WHAT runs; this decides which part of it runs HERE. | |
| # Round-robin by position, which is what spreads the expensive members: | |
| # opencv-module / -dnn / -unifont are adjacent in the list, so `% N` | |
| # necessarily puts them on three different runners. A single job that | |
| # builds all three spends 45+ minutes on opencv alone. | |
| # ── This shard's slice ──────────────────────────────────────────── | |
| # Already decided by `select` (measured-time bin packing, see | |
| # tests/plan_shards.lua). Arrives as data, so a runner needs no lua. | |
| - name: Take this shard's members | |
| shell: bash | |
| run: | | |
| mine='${{ fromJSON(needs.select.outputs.plan)[matrix.platform][format('{0}', matrix.shard)] }}' | |
| echo "MEMBERS=$mine" >> "$GITHUB_ENV" | |
| echo "shard ${{ matrix.shard }}/${{ matrix.shards }}: ${mine:-<none>}" | |
| # ── Refresh the PUBLISHED index before testing ──────────────────── | |
| # Most members resolve everything from this checkout, but a member that | |
| # redirects a namespace other than `compat` gets the REST from the | |
| # published index — and nothing here ever refreshed it. The snapshot in | |
| # play is whatever the pinned mcpp release vendored (the Download step | |
| # `cp -a`s the release's registry/ over ~/.mcpp/registry, on top of the | |
| # restored cache), so it is by construction older than main, and it | |
| # never moves: the cache is saved with that same stale copy inside it. | |
| # | |
| # mcpp does refresh on a miss for a DIRECT dependency, which is why this | |
| # went unnoticed — the gap is a Form-A package's TRANSITIVE dependency. | |
| # tests/examples/godot-cpp-module hit it head-on: the module package's | |
| # own compat.godot-cpp dep resolved against a snapshot predating the | |
| # commit that added it, and failed with `index: local index <sha> (never | |
| # refreshed)` even though the artifact had already been republished. | |
| # (Older members never noticed: their compat packages have been in the | |
| # index far longer than any snapshot.) | |
| - name: Refresh the published package index | |
| shell: bash | |
| env: | |
| MCPP_INDEX_MIRROR: GLOBAL | |
| run: | | |
| "$MCPP" index update | |
| - name: mcpp test (workspace or affected members) | |
| shell: bash | |
| env: | |
| MCPP_INDEX_MIRROR: GLOBAL | |
| # The GLOBAL package build cache is on (mcpp >= 2026.7.30.2), which | |
| # is the default — this step used to set `MCPP_BUILD_CACHE: local` | |
| # and no longer does. | |
| # | |
| # That bypass existed for mcpp#344: object-path disambiguation fires | |
| # on basename collisions across the WHOLE build dir — i.e. on what | |
| # the CONSUMER pulls in — while the cache key covered only the | |
| # dependency, so one entry could hold two layouts and ninja died at | |
| # graph time with "missing and no known rule to make it". #344 | |
| # landed in 2026.8.3.4 with per-package Merkle keys that cover the | |
| # consumer-dependent layout, so the reason is gone. | |
| # | |
| # Keeping it cost real time, and the full run is where it showed: | |
| # with `local`, EVERY member recompiles EVERY dependency from | |
| # scratch. 59 members that mostly share abseil / protobuf / opencv | |
| # meant the same sources were built over and over — | |
| # | |
| # linux 2h30m -> cancelled at the 150-minute timeout | |
| # windows 2h20m | |
| # macos 1h26m | |
| # | |
| # — and a workspace cannot be validated by a job that cannot finish. | |
| # With the cache on, a given (package, version, features, toolchain) | |
| # is built once per run and every later member hits it. | |
| run: | | |
| "$MCPP" --version | |
| # No `timeout` wrapper: absent on macOS runners; job-level timeout-minutes bounds it. | |
| # One code path: the shard step above already expanded `__ALL__` | |
| # into this runner's actual member names, so `mcpp test --workspace` | |
| # — which would ignore the sharding and rebuild everything here — is | |
| # gone. | |
| # | |
| # tests/run_members.sh is the SAME script you run locally. A timing | |
| # table that only exists in CI cannot be used while deciding what to | |
| # optimise, and a local harness that differs from CI measures | |
| # something else. | |
| if [ -z "$MEMBERS" ]; then | |
| echo "No workspace member affected by this change — nothing to test." | |
| else | |
| MCPP_TIMINGS="$PWD/timings.tsv" bash tests/run_members.sh $MEMBERS | |
| fi | |
| # Per-shard timings, merged by the `timings` job below. `always()`: a | |
| # run that failed is exactly when knowing where the time went matters. | |
| - name: Upload this shard's timings | |
| if: always() && hashFiles('timings.tsv') != '' | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| # The toolchain is part of the name because upload-artifact@v4 | |
| # refuses duplicates: without it both linux legs would race to | |
| # upload `timings-linux-0` and the second one would fail the job. | |
| name: timings-${{ matrix.platform }}-${{ matrix.toolchain }}-${{ matrix.shard }} | |
| path: timings.tsv | |
| retention-days: 14 | |
| # install()-driven packages (openssl, openblas) build through their own | |
| # Make/Configure system, whose output xim's interface mode swallows; a | |
| # failed hook surfaces only as `E_INTERNAL: [<pkg>] failed:`. Each writes | |
| # a log into its install prefix, so on failure surface those — otherwise | |
| # diagnosing a platform-specific build break costs a full CI round-trip | |
| # per guess. | |
| - name: Dump install() build logs on failure | |
| if: failure() | |
| shell: bash | |
| run: | | |
| found=0 | |
| while IFS= read -r log; do | |
| found=1 | |
| echo "::group::$log" | |
| tail -80 "$log" | |
| echo "::endgroup::" | |
| done < <(find tests/examples "$HOME/.mcpp/registry" -name 'mcpp_*_build.log' 2>/dev/null) | |
| [ "$found" = 1 ] || echo "no install() build logs found" | |
| # ── Where the time went ─────────────────────────────────────────────── | |
| # Sharding hides the cost: eight runners each report their own slice, and | |
| # nobody can see which members actually dominate. This merges them into one | |
| # ranking per platform, in the run summary, so the next optimisation starts | |
| # from measurement instead of a guess. | |
| # | |
| # `always()` — a failed run is exactly when this is worth reading. | |
| timings: | |
| needs: [select, workspace] | |
| if: always() && needs.select.outputs.members != '' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/download-artifact@v4 | |
| with: | |
| pattern: timings-* | |
| path: timings | |
| continue-on-error: true | |
| - name: Rank members by wall-clock | |
| shell: bash | |
| run: | | |
| shopt -s nullglob | |
| files=(timings/*/timings.tsv) | |
| if [ ${#files[@]} -eq 0 ]; then | |
| echo "no timing data (every shard skipped or failed before testing)" \ | |
| >> "$GITHUB_STEP_SUMMARY" | |
| exit 0 | |
| fi | |
| # Artifact name carries platform AND toolchain leg: | |
| # timings-<platform>-<toolchain>-<shard>. Ranking per LEG, not per | |
| # platform: the two linux legs are different builds and averaging | |
| # them would describe neither. | |
| for leg in linux-default linux-llvm macos-default windows-default; do | |
| rows=$(mktemp) | |
| for f in timings/timings-$leg-*/timings.tsv; do | |
| [ -f "$f" ] && cat "$f" >> "$rows" | |
| done | |
| [ -s "$rows" ] || { rm -f "$rows"; continue; } | |
| total=$(awk -F'\t' '{s += $1} END {print s+0}' "$rows") | |
| count=$(wc -l < "$rows") | |
| { | |
| echo "### $leg — ${count} member(s), ${total}s of member wall-clock" | |
| echo | |
| echo "| rank | seconds | share | member | result |" | |
| echo "|---:|---:|---:|---|---|" | |
| sort -rn "$rows" | awk -F'\t' -v tot="$total" ' | |
| { pct = tot > 0 ? ($1 * 100 / tot) : 0 | |
| printf "| %d | %s | %.1f%% | `%s` | %s |\n", NR, $1, pct, $2, $3 }' | |
| echo | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| rm -f "$rows" | |
| done | |
| echo "_Total is the SUM across shards; wall-clock is the slowest shard._" \ | |
| >> "$GITHUB_STEP_SUMMARY" | |
| # The table that feeds the NEXT run's sharding. Emitted as an | |
| # artifact rather than committed automatically: a number that | |
| # rewrites itself on every run would make every diff noisy and would | |
| # silently absorb a one-off slow runner. Refresh it deliberately — | |
| # download this artifact and replace tests/member-timings.tsv when | |
| # the numbers have actually moved. | |
| # Top 10 per platform, not every member. | |
| # | |
| # plan_shards prices anything absent at a fixed default, and the | |
| # packing decisions are made by the heavy members anyway — on linux | |
| # the top 10 of 67 are 69% of the total, and dropping the other 57 | |
| # moves the slowest shard by 0 minutes. A table that lists everyone | |
| # goes stale the moment someone adds a test; one that lists ten only | |
| # goes stale when the heavy set actually changes. That is the whole | |
| # point: adding a member must not require touching this file. | |
| { | |
| echo "# <platform>\t<member>\t<seconds> — the HEAVY members only, top 10" | |
| echo "# per platform, from run ${{ github.run_id }}. Everything else is" | |
| echo "# priced at plan_shards.lua's fixed default; adding a test does NOT" | |
| echo "# require touching this file. Refresh when a member becomes heavy" | |
| echo "# enough to enter the top 10, or one of these numbers moves." | |
| # ONLY the default-toolchain leg feeds this table. The llvm leg | |
| # runs the SAME members again, so globbing every leg would put two | |
| # rows per (platform, member) into the file — `sort -u` keeps both, | |
| # since the seconds differ. Both legs are planned from these | |
| # numbers, so the default leg is the right single baseline. | |
| # | |
| # Then: heaviest first, keep 10. `sort -u` before that so a member | |
| # reported by two shards cannot occupy two of the ten slots. | |
| for plat in linux macos windows; do | |
| for f in timings/timings-$plat-default-*/timings.tsv; do | |
| [ -f "$f" ] || continue | |
| awk -F'\t' -v p="$plat" '{ printf "%s\t%s\t%s\n", p, $2, $1 }' "$f" | |
| done | sort -u | sort -t"$(printf '\t')" -k3,3nr | head -10 | |
| done | |
| } > member-timings.tsv | |
| echo "wrote member-timings.tsv ($(grep -vc '^#' member-timings.tsv) rows, top 10/platform)" | |
| - name: Upload the timing table for the next run's sharding | |
| if: always() && hashFiles('member-timings.tsv') != '' | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: member-timings | |
| path: member-timings.tsv | |
| retention-days: 90 |