Skip to content

ci: 刷新计时表并把 linux 分片上限抬到 4 —— 3 片已经装不下了 #664

ci: 刷新计时表并把 linux 分片上限抬到 4 —— 3 片已经装不下了

ci: 刷新计时表并把 linux 分片上限抬到 4 —— 3 片已经装不下了 #664

Workflow file for this run

name: validate
on:
pull_request:
# mcpp.toml and index.toml carry the workspace member list, the inherited
# [indices] redirect and the client version floor — a change to any of them
# can break every member, so they gate the run like the descriptors do.
paths: ["pkgs/**/*.lua", "tests/**", "README.md", "README.zh-CN.md", "mcpp.toml", "index.toml", ".github/workflows/validate.yml"]
push:
branches: [main]
schedule:
# nightly full regression — exercises every workspace member regardless of diff
- cron: "0 6 * * *"
workflow_dispatch:
inputs:
cache:
description: "Package build cache — 'local' rebuilds every dependency per member, which is what the timing table should be read against when comparing"
type: choice
options: [global, local]
default: global
env:
# 2026.8.5.4 carries two things this workflow depends on:
# .5.1 `tools = [...]` — how a consumer asks for a dependency's
# `kind = "bin"` target, which tests/examples/protobuf-protoc is built
# on. Before it: "tools must be a string, inline dep table, or nested
# table".
# .5.4 windows links with lld. link.exe caps a response-file LINE at
# 128 KiB and opencv-module / opencv-module-dnn went past it —
# fatal error LNK1170: line in command file contains 135135 or
# more characters
# after 795s / 1166s of compiling. .5.3 newline-separated OUR response
# file, which was necessary but not sufficient: clang, acting as the
# driver, writes a SECOND one for the linker that we do not control.
# lld's response-file parser has no per-line limit at all.
# Together with re-enabling the global package cache below, this is
# what makes a green FULL run possible again: .5.3 removes the
# windows link failure, the cache removes the 150-minute timeout.
#
# Neither of them moves index.toml's min_mcpp: exposing compat.protobuf's `protoc`
# target is additive, and 2026.8.3.3 still parses that descriptor with an
# empty unknown_keys. The floor an index publishes decides whether older
# clients keep working at all (mcpp#349), so it moves only when a descriptor
# genuinely stops being readable — which is not the case here.
# 2026.8.3.1: on macOS, a global object that touches std::cout during static
# init crashes on sight (mcpp#336). Mach-O has no priority-ordered init
# section and libc++'s <iostream> carries no ios_base::Init guard of its own,
# so the streams are still all-zero when an archive member's initializer
# runs. mcpp now links a generated object FIRST whose constructor brings them
# up. It is not fixable package-side — std::ios_base::Init is only
# forward-declared in libc++'s <ios> — so boost-ext.ut genuinely requires
# this floor on macOS, and min_mcpp/latest_mcpp move with the pin as they
# always have (a consumer below the floor would get a segfault with no
# diagnostic, which is worse than E0006).
# 2026.8.3.3 is the pin rather than .3.1: .3.2/.3.3 are cross-compilation
# fixes (PE artifact naming, -static host-vs-target) that no leg of this
# matrix exercises, so taking the newest of the train costs nothing.
# 0.0.109: a bare dependency's wire address takes BOTH halves from the
# descriptor the identity gate accepted (mcpp#286). This is the client-side
# other half of the SPEC-001 migration below: mcpp used to take the NAME from
# the descriptor and the NAMESPACE from the request, so a bare `gtest =
# "1.15.2"` addressed `mcpplibs:gtest` — a key no index has. It only ever
# worked because the pre-migration literal `package.name` read
# "compat.gtest", which the hardcoded `compat.<short>` retry then caught.
# Short names removed that coincidence and left every bare request against
# this index broken on 0.0.108, which is why min_mcpp moves to 0.0.109.
# Note this index cannot cover that spelling itself: the `[indices]` redirect
# is keyed by the REQUEST's namespace, so a bare dependency resolves from the
# published remote index rather than the checkout under test. Bare-name
# resolution against a short-name index is upstream's e2e 165; what moving
# the floor buys here is that consumers of THIS index get a client that can
# address it.
# 0.0.106: SPEC-001 package identity (mcpp#280). `package.name` is a SINGLE
# ATOMIC SEGMENT — all hierarchy lives in `package.namespace` — and mcpp
# addresses a package by the LITERAL name it read, so descriptors no longer
# repeat their namespace inside `name`. This index is migrated to the short
# form, which is why min_mcpp/latest_mcpp move in lock-step: an older client
# re-derives `<ns>.<short>`, misses, and reports a bare E_NOT_FOUND. Bundles
# xlings 0.4.69, which keys its index by (namespace, name) so two packages
# sharing a short name in one index are both addressable (xlings#381) — this
# index now has three such pairs (imgui / ffmpeg / lua under compat vs the
# default namespace).
# 0.0.102: windows command-line ceiling (mcpp#261 — the clang scan rule got
# its P1689 JSON through shell redirection, which forced a `cmd /c` wrapper
# and with it cmd.exe's 8191-char limit; clang-scan-deps -o removes both, and
# $local_includes-carrying rules now fall back to response files). The pin
# matters here because a package consumed FROM the registry sits under a
# ~124-char xpkgs path instead of its own ~23-char checkout, which is what
# pushed the vendored-opencv scan command over the line. Also: purview-include
# depfile tracking extended to Clang (#257 — stale BMI reuse), OS-conditional
# `[build].flags` (#258), and per-OS splices keyed on the resolved target
# rather than the host (#254).
# 0.0.101: per-feature per-glob flags + per-OS features (mcpp#253) — what
# lets opencv select its dnn gemm backend per platform.
# 0.0.99: feature dep/feat forwarding (mcpp#243 — a feature can open a
# feature OF a dependency, e.g. opencv `dnn` forwarding compat.opencv/dnn);
# vendored xlings 0.4.67 for the >=2 index_repo install fix (mcpp#238 /
# openxlings/xlings#374); build.mcpp compiled program named `.exe` on
# Windows (mcpp#230 secondary surface, after the 0.0.96 scanner crash fix).
# 0.0.98: closes the obj-path disambiguation follow-ups that gated the
# source-build compat.opencv unification — #240 (link inputs now follow
# the disambiguated object names, so a dependency + consumer sharing a
# source basename like `src/main.cpp` no longer 'obj/main.o missing') and
# #239 (absolute/`..` dep-generated source paths sanitized component-wise
# so objects stay under obj/). Also: `MCPP_DEP_<NAME>_DIR` build.mcpp
# contract (#241), consumer-side `default-features = false` (#242), and a
# loud unknown-mcpp-key warning with did-you-mean (#237, replaces the
# silent-ignore at build time). Carried from 0.0.97: default-namespace
# index redirect (`[indices] default = { path }`), which turned the public
# module packages (imgui/ffmpeg/opencv/tinyhttps) into ordinary workspace
# members and retired the per-package reseeding smoke shells + their
# dedicated jobs; synchronous nasm bootstrap (mcpp#232 — the `mcpp index
# update` pre-step is gone), obj-path disambiguation (#233), spacey-defines
# quoting (#234), purview-include depfile tracking (#235). Older floors of
# note: 0.0.96 fixed the windows scanner symlink-escape crash (mcpp#230);
# 0.0.94 fixed feature-gated `sources` under `mcpp test` (mcpp#218); 0.0.91
# added standard = "c++fly" to the resolver grammar, so c++fly descriptors
# get the lint WARN below, not a hard grammar-parse rejection.
# 2026.8.6.1 是本次身份迁移的**前置**,不是顺手升级。
#
# `Fetcher::install_path(ns, shortName, version)` 的 legacy 扫描此前匹配任何以
# `-x-<shortName>` 结尾的目录、不看命名空间,于是查 `ocornut:imgui@1.92.8` 会拿到
# `compat-x-imgui/1.92.8` —— 另一个仅仅短名相同的包。它一直够不到,是因为 module
# 层用打包计数(imgui@0.0.6)而 compat 用上游版本(compat.imgui@1.92.8),版本永远
# 不撞;本次把 module 层对齐到上游之后它们重合了。mcpp#364 修掉了它。
#
# index.toml 的 min_mcpp **不动**。两个会撞的 compat 邻居都是 Form B,所以旧客户端
# 撞上时是响亮报错而不是静默用错包;下限是一道让整个索引对旧客户端失效的闸门
# (mcpp#349),只该在描述符真的读不动时抬。这里读得动,差的是解析得对。
#
# 2026.8.8.2 是本 PR 的**前置**,不是顺手升级。
#
# `compat.glx-runtime` 依赖 mesa,mesa 声明 `xim:glibc@>=2.38`;下限被任何更高
# 版本满足,于是安装图形栈会在既有 glibc 旁边**再装一个**。2026.8.8.2 之前的
# mcpp 用 `readdir` 的第一项来解析「那个 glibc payload」,编译侧与产物的
# interpreter 因此可以指向不同版本 —— 这正是 #179 落地后 `asio-module` 和
# `core` 变红、并导致整份改动被 #180 revert 的原因。
#
# 用早于 2026.8.8.2 的 mcpp 重新落地这份改动,就是在仍会犯这个错的引擎上复现
# 事故条件。tests/check_graphics_install_side_effects.sh 也需要它 —— 该测试在
# 更旧的 mcpp 上失败是**有意的**。
MCPP_VERSION: "2026.8.8.2"
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install lua
run: sudo apt-get install -y --no-install-recommends lua5.4
- name: Lint package descriptors
run: |
fail=0
for f in pkgs/*/*.lua; do
# 1. Lua syntax check — load (= compile) without executing.
# `loadfile(name, 't')` rejects bytecode and parses text only.
if ! lua5.4 -e "assert(loadfile('$f', 't'))" >/dev/null 2>&1; then
echo "::error file=$f::lua syntax error"
fail=1
fi
# 2. xpkg V1 baseline: the file has to populate `package = { ... }`
# with at least `spec`, `name`, and an `xpm` table. Form A vs
# Form B (mcpp = "<path>" / mcpp = { ... }) is descriptor-author
# choice and not enforced here.
for needle in 'spec *=' 'name *=' 'xpm *='; do
if ! grep -q "$needle" "$f"; then
echo "::error file=$f::missing required field ($needle)"
fail=1
fi
done
# 3. Package version identifiers and dependency versions should be
# bare versions ("1.2.3"), not upstream tag names ("v1.2.3").
# Download URLs may still contain refs/tags/v* when upstream
# uses that tag spelling.
if grep -nE '\["v[0-9]+|\["[^"]+"\][[:space:]]*=[[:space:]]*"v[0-9]+' "$f"; then
echo "::error file=$f::version identifiers must not use a leading v"
fail=1
fi
# 4. Mirror table sanity: when a download `url` is written as a
# { GLOBAL=..., CN=... } table, both regions must be present and
# the CN entry must point at the gitcode mcpp-res mirror.
if ! lua5.4 tests/check_mirror_urls.lua "$f"; then
fail=1
fi
# 5. `name` must be a SINGLE ATOMIC SEGMENT; hierarchy belongs in
# `namespace` (mcpp SPEC-001 §3.2). The legacy fully-qualified
# spelling stays accepted. Cheap second gate: it runs before the
# pinned mcpp is even downloaded, and mcpp >= 0.0.106 enforces
# the same rule inside `mcpp xpkg parse`.
if ! lua5.4 tests/check_package_name.lua "$f"; then
fail=1
fi
# 6. c++fly admission policy (mcpp design 2026-07-14 §11-Q2, v1):
# c++fly means "toolchain's latest level + every experimental
# gate" — deliberately toolchain-dependent, so a published
# package built with it is not reproducible for consumers.
# Policy: WARN (never fail) and observe ecosystem usage before
# deciding whether to tighten. Two spellings: `language = ` is
# the descriptor's inline mcpp-segment key; `standard = ` covers
# mcpp.toml content embedded in heredoc/generated_files blocks.
if grep -nE '\b(language|standard)[[:space:]]*=[[:space:]]*"c\+\+fly"' "$f" >/dev/null; then
echo "::warning file=$f::declares C++ standard \"c++fly\" (experimental playground mode) — toolchain-dependent and non-reproducible for consumers; published packages should pin a concrete standard (c++23/c++26)"
fi
done
[ $fail -eq 0 ] && echo "All package files valid."
exit $fail
# ── Whole-repository check (needs every descriptor at once) ──────
# An install() hook addressing a sibling package does so by
# `<namespace>:<literal package.name>`, and a miss returns nil rather
# than raising — so a stale spelling surfaces far downstream (a broken
# libxcb showed up as a libX11 link error). Verified across the repo
# because it needs the full set of declared identities.
- name: Lint cross-package references
run: lua5.4 tests/check_cross_package_refs.lua pkgs/*/*.lua
# ── Partial version bumps ────────────────────────────────────────
# A bump is a one-line-looking edit that has to land in N platform
# sections. Editing `xpm.linux` and reading the file back gives a file
# that CONTAINS the new version, so the author -- and any whole-file
# grep -- sees success, while the other platforms are left behind. The
# failure then surfaces as `<pkg>@<ver> not found` on a platform,
# against a file that literally contains that version string.
#
# Measured 2026-08-06: xpkg 0.0.52 and 0.0.53 were both added to
# `xpm.linux` alone; linux CI went green twice while macOS and Windows
# failed, and eight checks made from the outside all came back correct
# because each asked "is it in the index?" instead of "is it in THIS
# platform's section?". Whole-repo, because a partial bump is only
# visible by comparing sections against each other.
- name: Lint platform version parity
run: lua5.4 tests/check_platform_version_parity.lua pkgs/*/*.lua
# ── Single-source-of-truth grammar check ─────────────────────────
# `mcpp xpkg parse` uses EXACTLY the resolver's parser, so what
# passes here is what builds for users of the pinned MCPP_VERSION.
# Strict by default: unknown mcpp-segment keys fail (they would be
# silently ignored at build time). This also mechanically enforces
# the rollout rule "floor first, new grammar after": descriptors
# needing a newer grammar cannot pass a lint pinned to an older mcpp.
- name: Download pinned mcpp
run: |
curl -L -fsS -o mcpp.tar.gz \
"https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/mcpp-${MCPP_VERSION}-linux-x86_64.tar.gz"
tar -xzf mcpp.tar.gz
echo "MCPP=$PWD/mcpp-${MCPP_VERSION}-linux-x86_64/bin/mcpp" >> "$GITHUB_ENV"
- name: Parse descriptors with the resolver grammar (mcpp xpkg parse)
run: |
fail=0
for f in pkgs/*/*.lua; do
if ! "$MCPP" xpkg parse "$f" > /dev/null; then
echo "::error file=$f::mcpp xpkg parse failed (resolver grammar)"
fail=1
fi
done
[ $fail -eq 0 ] && echo "All descriptors parse with mcpp ${MCPP_VERSION}."
exit $fail
mirror-cn-reachable:
# Closed-loop guard for the CN mirror: every CN url referenced by a
# descriptor must be a live, downloadable gitcode release asset.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install lua
run: sudo apt-get install -y --no-install-recommends lua5.4
- name: Check CN mirror assets are reachable
run: |
fail=0
# collect unique CN urls across all descriptors
: > /tmp/cn.tsv
for f in pkgs/*/*.lua; do
lua5.4 tests/list_cn_urls.lua "$f" >> /tmp/cn.tsv || true
done
sort -u /tmp/cn.tsv -o /tmp/cn.tsv
total=$(grep -c . /tmp/cn.tsv || true)
echo "checking $total CN mirror url(s)"
while IFS=$'\t' read -r url sha; do
[ -z "$url" ] && continue
# follow redirects; gitcode release assets resolve to object storage
code=$(curl -fsSL -o /dev/null -w '%{http_code}' --retry 2 --max-time 60 "$url" || echo "000")
if [ "$code" != "200" ]; then
echo "::error::CN mirror unreachable ($code): $url"
fail=1
else
echo "ok: $url"
fi
done < /tmp/cn.tsv
[ $fail -eq 0 ] && echo "All CN mirror urls reachable."
exit $fail
# ── The whole test surface, as a mcpp workspace ───────────────────────
# mcpp-index is a mcpp [workspace]; every per-library test project under
# tests/examples/ is a member. `mcpp test --workspace` builds + runs each
# member's tests/ (behavioral assertions) on each OS — members self-gate by
# `[target.'cfg(...)']` (e.g. the X11/glfw stack is linux-only, openblas is
# windows-only), so one command covers the matrix with no shell driver.
# The ~/.mcpp/registry cache carries the built compat packages (xpkgs) across
# runs, so repeat builds are fast.
#
# timeout-minutes is sized for the COLD build, not the cached path. The opencv
# module package carries a from-source OpenCV 5 build, and each feature variant
# re-keys the store into a full recompile, so a full run (forced whenever this
# workflow file changes — e.g. a version bump) serially builds three OpenCV
# variants on one runner: the opencv-module base member plus the `unifont` and
# `dnn` feature members. The registry cache (restore-keys prefix below)
# amortizes those across subsequent runs. 150 covers the one-time cold full
# build with headroom; it is a ceiling, not a target.
# ── The plan, computed ONCE ───────────────────────────────────────────
# Was inlined in every workspace job — three runners each re-deriving the
# same answer. It now also has to be decided BEFORE the matrix exists,
# because the matrix's shard dimension depends on it: a full run fans out,
# a selective one does not.
select:
runs-on: ubuntu-latest
outputs:
members: ${{ steps.fanout.outputs.members }}
matrix: ${{ steps.fanout.outputs.matrix }}
plan: ${{ steps.plan_shards.outputs.plan }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install lua
run: sudo apt-get install -y --no-install-recommends lua5.4
# ── Selective member testing ──────────────────────────────────────
# `mcpp test --workspace` builds every member (opencv, ffmpeg, …) and
# dominates CI wall-clock, while a PR almost always touches one
# package. Map changed files → affected members and test only those:
# pkgs/<x>/<lib>.lua → members whose mcpp.toml references <lib>
# tests/examples/<m>/** → member <m>
# Run the FULL workspace when the change can affect everything:
# non-PR events (push to main, the nightly cron, dispatch), this
# workflow file (it carries the mcpp version pins, so a version bump
# always re-validates every package), a non-member edit to the
# workspace manifest, or shared test scripts. Docs-only and tools/-only
# changes select nothing.
# Note: bash 3.2 on macOS runners — no associative arrays here.
- name: Select affected workspace members
id: plan
shell: bash
run: |
full() { echo "MEMBERS=__ALL__" >> "$GITHUB_ENV"; echo "full run: $1"; exit 0; }
[ "${{ github.event_name }}" = "pull_request" ] || full "event=${{ github.event_name }}"
base="origin/${{ github.base_ref }}"
changed=$(git diff --name-only "$base"...HEAD)
printf 'changed files vs %s:\n%s\n' "$base" "$changed"
sel=""
add() { case " $sel " in *" $1 "*) ;; *) sel="$sel $1" ;; esac; }
while IFS= read -r f; do
[ -n "$f" ] || continue
case "$f" in
.github/workflows/validate.yml|tests/*.sh) full "$f" ;;
mcpp.toml)
# Workspace manifest. Every new-package PR appends to the
# members list, so that alone must NOT force a full run:
# select the added members; anything else in this file
# (indices, settings) affects everyone → full.
if ! diff -q <(git show "$base:mcpp.toml" | grep -v 'tests/examples/') \
<(grep -v 'tests/examples/' mcpp.toml) >/dev/null; then
full "mcpp.toml non-member change"
fi
for p in $(comm -13 <(git show "$base:mcpp.toml" | grep -o 'tests/examples/[A-Za-z0-9._-]*' | sort -u) \
<(grep -o 'tests/examples/[A-Za-z0-9._-]*' mcpp.toml | sort -u)); do
add "${p#tests/examples/}"
done ;;
tests/examples/*)
m=${f#tests/examples/}; m=${m%%/*}
# A deleted/renamed member dir implies a mcpp.toml edit,
# which already forces a full run above.
[ -d "tests/examples/$m" ] && add "$m" ;;
pkgs/*.lua|pkgs/*/*.lua)
lib=$(basename "$f" .lua); lib=${lib#compat.}
hit=0
for mt in tests/examples/*/mcpp.toml; do
if grep -q "$lib" "$mt"; then add "$(basename "$(dirname "$mt")")"; hit=1; fi
done
[ "$hit" = 1 ] || echo "note: no workspace member exercises $f" ;;
# tools/ holds OFFLINE descriptor-generation and publishing
# helpers (tools/compat-*/, tools/gtc/, publish_mcpp_index.sh).
# Nothing under it is consumed by a package build: when one of
# them actually changes a package, the generated pkgs/*.lua
# changes with it and the rule above selects the right members.
# So a tools/ edit alone selects nothing rather than forcing a
# full workspace rebuild.
# The timing table decides how work is DISTRIBUTED across
# shards, never what gets built — no member's result can change
# because a measured number moved. It used to fall through to
# the `*)` catch-all below and force a full run, which is the
# most expensive way in this workflow to test nothing: the next
# full run reads the new numbers anyway.
#
# Note this makes the file invisible to CI. A malformed row is
# then silent, and `plan_shards` prices anything it cannot parse
# at the median — the exact failure mode that overflowed a shard
# here. Guard it in `lint` if that ever bites.
tests/member-timings.tsv) : ;;
*.md|docs/*|.agents/*|.github/*|tools/*) : ;;
*) full "unclassified change: $f" ;;
esac
done <<EOF
$changed
EOF
sel=${sel# }
echo "MEMBERS=$sel" >> "$GITHUB_ENV"
echo "selected members: ${sel:-<none>}"
# Sharding is for the FULL run only, and the shard count per platform is
# that platform's RUNNER CONCURRENCY — not a round number.
#
# Measured on this repo (24 jobs queued, 6 running):
# macos 1 · linux 3 · windows 2
#
# That measurement is what makes over-sharding a real cost rather than a
# theoretical one: at concurrency 1, eight macOS shards run BACK TO BACK
# and each pays its own checkout + mcpp download + cache restore, so the
# split is strictly slower than not splitting. Wall-clock is
# ceil(shards / concurrency) x slowest-shard; shards beyond the
# concurrency only add fixed cost.
#
# Re-measure with:
# gh api repos/<owner>/<repo>/actions/runs/<id>/jobs --paginate \
# --jq '[.jobs[]|select(.status=="in_progress")]|length'
- name: Decide the fan-out
id: fanout
shell: bash
run: |
full=0; [ "$MEMBERS" = "__ALL__" ] && full=1
emit() { # platform os suffix ext mcpp xlings toolchain shards
for i in $(seq 0 $(( $8 - 1 ))); do
printf '{"platform":"%s","os":"%s","suffix":"%s","ext":"%s","mcpp":"%s","xlings":"%s","toolchain":"%s","shard":%d,"shards":%d},' \
"$1" "$2" "$3" "$4" "$5" "$6" "$7" "$i" "$8"
done
}
# Shard count follows the WORK, per platform, measured.
#
# It used to be binary — full run: linux 3 / macos 1 / windows 2,
# anything else: 1 each — which asserts two things that are not
# true. "A partial run is small": touching a widely consumed
# descriptor selects every member that consumes it, and this PR's
# `pkgs/c/compat.openssl.lua` selected four; one linux shard took
# grpc-codegen (3563s) and grpc-module (1701s) back to back and was
# cancelled at exactly 1h30m, the job cap. "A full macOS run fits in
# one shard": at 6922s measured it does not — it fit only while the
# total sat just under the cap, and the run that added grpc-codegen
# (1715s on macOS) pushed it to 1h30m20s.
#
# Sharding is not only a wall-clock lever. Each shard is its own job
# with its own `timeout-minutes`, so it is also how the work is made
# to FIT. That is the half the "macOS concurrency is 1, so extra
# shards run back to back" note above left out: back to back is fine
# when the alternative is not finishing.
#
# tests/member-timings.tsv already holds every member's measured
# cost and plan_shards.lua already packs by it (LPT), so the fan-out
# comes from the same table: sum this run's members for that
# platform and take one shard per ~70 minutes, which leaves ~20
# minutes of headroom under the 90-minute cap for a cold cache.
#
# The caps are where runner concurrency comes back in: linux 4,
# macOS 2, windows 2. Past those, more shards buy fit that is
# already there and pay another checkout + mcpp download + cache
# restore.
#
# linux was 3 — "the measured concurrency, a 4th shard would queue".
# That cap outlived its premise twice over: the workspace grew, and
# the second toolchain leg means linux emits 2 x N jobs, so it
# queues at any N. It is now 4, which is what the formula asks for.
#
# Full run, from the table: linux 15891s over 4 -> ~74min slowest
# shard, macOS 6922s over 2 -> ~57, windows 8043s over 2 -> ~67.
#
# linux is still the tight one, and the number to watch is the
# SLOWEST shard against the 90-minute job cap, not the average.
# Simulated on the current table: 3-way 98min, 4-way 74, 5-way 60.
# The levers, in order: raise the linux cap to 5, then the job
# timeout.
#
# Keep tests/member-timings.tsv fresh — that is not housekeeping. An
# untimed member is priced at the MEDIAN, so a heavy newcomer packs
# like a trivial one. `mysql-connector-cpp` (881s) and
# `libmysqlclient` (329s) landed on shard 0 next to grpc-codegen
# exactly that way and pushed it past the cap, and the stale table
# then hid the overflow by under-counting the total.
# `plan_shards.lua <platform> 0 0` reports the platform's total cost.
#
# This used to sum the timing table's rows inline, which was the same
# answer only while the table listed every member. It now lists just
# the heavy ones, so summing rows measures a fraction of the work:
# 10965s instead of 16095s on linux, i.e. 3 shards where 4 are
# needed, i.e. a slowest shard of 107 minutes against a 90-minute
# cap. Asking plan_shards keeps ONE definition of both "which members
# are there" and "what does an untimed one cost".
#
# The estimate is tuned on linux (16095s vs 15891s measured, +1.3%).
# macOS and windows members are cheaper on average than the default,
# so their totals come out high — harmless here because both are
# already at their cap, and erring toward more shards is the safe
# direction. Revisit if either cap moves.
shards_for() { # platform cap -> shard count
local secs sel=""
[ "$MEMBERS" = "__ALL__" ] || sel="$MEMBERS"
secs=$(lua5.4 tests/plan_shards.lua "$1" 0 0 $sel)
local n=$(( secs / 4200 + 1 ))
[ "$n" -gt "$2" ] && n="$2"
echo "$1 work: ${secs}s (measured) -> $n shard(s)" >&2
echo "$n"
}
# linux 3 -> 4. The cap was the binding constraint, not the ~70min
# target: `secs / 4200 + 1` asks for 4 on the refreshed table
# (15891s) and has been getting clamped back to 3. Simulated on that
# table, slowest shard: 3-way 98min (OVER the 90min job cap — and
# `linux default 0/3` was in fact cancelled at 90min on run
# 31266814148), 4-way 74min, 5-way 60min. 4 is the formula's own
# answer and leaves 16 minutes for a cold cache; 5 is there if that
# stops being enough.
#
# This does cross the concurrency line the old comment drew: with two
# toolchain legs linux now emits 8 jobs against a measured runner
# concurrency of 3, so shards queue. Queuing is the right trade —
# back-to-back shards still finish, a shard over the cap does not.
ln=$(shards_for linux 4)
mn=$(shards_for macos 2)
wn=$(shards_for windows 2)
{
printf '{"include":['
# linux runs TWICE: once on mcpp's default toolchain and once on
# llvm. Not redundancy — the two differ in a way that decides
# whether a descriptor is correct. mcpp's linux default (gcc)
# reaches the compiler through --sysroot into a clean xlings
# subos, so the HOST /usr/include is not on the search path;
# llvm has no sysroot and the host's headers are. compat.ffmpeg
# put its vendored source root on -idirafter (below the system
# dirs) and was green here for months while being broken on any
# developer box with libavutil-dev installed — the gcc leg
# structurally cannot see that class of bug.
#
# Cost, measured: this doubles linux to 6 jobs (2 x 3 shards) at
# ~75min each, against a measured linux runner concurrency of 3 —
# so the second leg QUEUES behind the first rather than running
# beside it, and a full run's linux wall-clock roughly doubles.
# That is the price of the coverage; the levers if it hurts are
# in the shards_for comment above.
emit linux ubuntu-latest linux-x86_64 tar.gz bin/mcpp registry/bin/xlings default "$ln"
emit linux ubuntu-latest linux-x86_64 tar.gz bin/mcpp registry/bin/xlings llvm "$ln"
emit macos macos-15 macosx-arm64 tar.gz bin/mcpp registry/bin/xlings default "$mn"
emit windows windows-latest windows-x86_64 zip bin/mcpp.exe registry/bin/xlings.exe default "$wn"
printf ']}'
} | sed 's/,]}/]}/' > /tmp/matrix.json
echo "matrix=$(cat /tmp/matrix.json)" >> "$GITHUB_OUTPUT"
echo "members=$MEMBERS" >> "$GITHUB_OUTPUT"
cat /tmp/matrix.json
# The split is computed ONCE, here, and shipped to the runners as data.
# It used to run on each runner, which needed lua5.4 on all three
# platforms — windows has no apt or brew, and macOS's brew installs
# `lua`, not `lua5.4`, so every non-linux shard died with
# `lua5.4: command not found` after 16 seconds. Deciding once is also
# simply correct: one plan, not three runners each re-deriving it.
- name: Plan the shards
id: plan_shards
shell: bash
run: |
plan='${{ steps.fanout.outputs.members }}'
[ "$plan" = "__ALL__" ] && plan=""
{
printf '{'
first=1
# `.shards`, NOT the number of matrix entries.
#
# These stopped being the same number the moment linux grew a
# second toolchain leg: the platform emits 2 x 3 = 6 entries while
# the split is still 3 ways, and every entry carries shard 0..2.
# Planning 6 ways and consuming three of them dropped 36 of 65
# members on the floor — silently, because a member that is never
# assigned is indistinguishable from one that passed. `ffmpeg`,
# `opencv-module*`, `catch2-v2` and `openssl` were among them, on
# the very run that added the leg meant to test them.
#
# `.shards` is the value emit() already wrote per entry and the
# same one the job reads for its own `matrix.shards`, so plan and
# consumer now read one number instead of two that agreed by
# accident.
for spec in linux:$(jq -r 'first(.include[]|select(.platform=="linux")).shards' /tmp/matrix.json) \
macos:$(jq -r 'first(.include[]|select(.platform=="macos")).shards' /tmp/matrix.json) \
windows:$(jq -r 'first(.include[]|select(.platform=="windows")).shards' /tmp/matrix.json); do
p=${spec%%:*}; n=${spec##*:}
[ "$first" = 1 ] || printf ','
first=0
printf '"%s":{' "$p"
for i in $(seq 0 $((n - 1))); do
[ "$i" = 0 ] || printf ','
m=$(lua5.4 tests/plan_shards.lua "$p" "$i" "$n" $plan)
printf '"%s":"%s"' "$i" "$m"
done
printf '}'
done
printf '}'
} > /tmp/plan.json
echo "plan=$(cat /tmp/plan.json)" >> "$GITHUB_OUTPUT"
jq . /tmp/plan.json
# Installing the graphics stack must not change what UNRELATED members link
# against. This is the check #179 did not have: every test in this repo asks
# about its own package, so nothing asked whether installing one changes
# things for everyone else -- and `asio-module` / `core` went red on a
# change that never touched them.
#
# Its own job, on Linux, because it needs a real toolchain and a home it can
# watch payloads appear in. It reports INCONCLUSIVE (and fails) rather than
# green whenever its preconditions do not hold; see the script.
graphics-side-effects:
name: "graphics install: no side effects on unrelated members"
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- name: Download mcpp
shell: bash
env:
MCPP_ARCHIVE: mcpp-${{ env.MCPP_VERSION }}-linux-x86_64.tar.gz
MCPP_ROOT: mcpp-${{ env.MCPP_VERSION }}-linux-x86_64
run: |
curl -L -fsS -o "$MCPP_ARCHIVE" \
"https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/${MCPP_ARCHIVE}"
tar -xzf "$MCPP_ARCHIVE"
root="$PWD/$MCPP_ROOT"
# MCPP_HOME is the tarball root, not ~/.mcpp.
#
# A released mcpp is self-contained: it resolves its registry from
# beside its own executable. Copying the payload tree into ~/.mcpp
# and pointing MCPP_HOME there therefore watches one home while the
# build uses another -- which this check detects and refuses to call
# green (it did, on the first run). Aligning them is the fix; the
# check was right.
echo "MCPP=$root/bin/mcpp" >> "$GITHUB_ENV"
echo "MCPP_HOME=$root" >> "$GITHUB_ENV"
echo "MCPP_VENDORED_XLINGS=$root/registry/bin/xlings" >> "$GITHUB_ENV"
- name: Side-effect check
shell: bash
run: |
"$MCPP" self config --mirror GLOBAL 2>/dev/null || true
bash tests/check_graphics_install_side_effects.sh
workspace:
# The shard suffix appears only when the platform is actually split, and
# the toolchain only when it is not mcpp's default — so the existing job
# names are unchanged and only the new llvm leg is labelled.
name: workspace (${{ matrix.platform }}${{ matrix.toolchain == 'default' && '' || format(' {0}', matrix.toolchain) }}${{ matrix.shards == 1 && '' || format(' {0}/{1}', matrix.shard, matrix.shards) }})
needs: select
if: needs.select.outputs.members != ''
runs-on: ${{ matrix.os }}
# One shard is a fraction of the work, so this is a real ceiling rather
# than the thing that decides whether the job finishes (a full linux run
# used to hit 150 exactly and get cancelled).
timeout-minutes: 90
strategy:
fail-fast: false
# Whole matrix from `select`: the shard count is per-platform, because it
# tracks that platform's runner concurrency.
matrix: ${{ fromJSON(needs.select.outputs.matrix) }}
steps:
# Full history: the member-selection step below diffs against the PR
# base to decide which workspace members to test.
- uses: actions/checkout@v4
with:
fetch-depth: 0
# The cache key is computed ONCE, here, instead of inline in the cache
# step. `hashFiles()` globs the WORKING TREE, and actions/cache
# re-evaluates its `key` in the post (save) step — i.e. AFTER the build,
# when `tests/**` no longer matches 80-odd tracked sources but tens of
# thousands of build-output files under tests/examples/*/target and
# .mcpp (multi-GB; .gitignore does not apply to hashFiles). Hashing that
# tree blew past the runner's 120s template-evaluation cap on windows
# and failed an otherwise all-green job:
# "hashFiles('pkgs/**/*.lua, tests/**, .github/workflows/validate.yml')
# couldn't finish within 120 seconds"
# `git ls-files -s` reads the INDEX, so it sees exactly the tracked
# inputs, never build output, and reports blob SHAs git already has —
# no file content is read at all. Freezing the result in the job env
# also guarantees the save step keys on the same string the restore
# step used, no matter what the build left behind.
- name: Compute registry cache key
shell: bash
run: |
# git hash-object rather than sha256sum/cut: git is already a hard
# requirement here (checkout ran), coreutils on the windows leg is
# only a Git-Bash convenience.
h=$(git ls-files -s -- 'pkgs/**/*.lua' 'tests/**' '.github/workflows/validate.yml' \
| git hash-object --stdin)
# matrix.toolchain is in the key: both linux legs are runner.os
# Linux, and the cache holds TOOLCHAINS and built compat packages.
# Sharing one entry would let the gcc leg's objects answer for the
# llvm leg — exactly the confusion this second leg exists to catch.
echo "REGISTRY_CACHE_KEY=mcpp-registry-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}-$h" >> "$GITHUB_ENV"
- name: Restore mcpp registry cache
uses: actions/cache@v4
with:
# Holds toolchains AND the built compat packages (data/xpkgs), so a
# repeat `mcpp test` rebuilds little.
path: ~/.mcpp/registry
key: ${{ env.REGISTRY_CACHE_KEY }}
restore-keys: |
mcpp-registry-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}-
# Host tools built from source (protoc, grpc_cpp_plugin, …).
#
# Measured on the run that added grpc-codegen: 636s to build protoc and
# 660s to build grpc_cpp_plugin — 1296s of a 3363s member, repeated in
# every job of every run, because the registry cache above covers
# ~/.mcpp/registry and the tool store lives in ~/.mcpp/build-cache.
# `protobuf-protoc` and `grpc-module` build the SAME protoc, so a full
# run pays for it several times over.
#
# A coarse rolling key is safe here, which is why this is cheap. The
# store is content-addressed by `<pkg>@<version>/<hash>` and mcpp
# validates an entry FIELD BY FIELD against the recorded entry.json
# (epoch, target, host triple, compiler identity, profile, features,
# transitive dependency closure). A stale entry is not used; it is
# rebuilt. The worst case of a cache miss-match is the status quo.
#
# Deliberately NOT build-cache/v1/pkg: 6.0 GB measured locally against
# Actions' 10 GB per-repo budget, which would evict the registry cache
# this job needs more. The tool store is 116 MB.
- name: Cache host tool store
uses: actions/cache@v4
with:
path: ~/.mcpp/build-cache/v1/tool
key: mcpp-toolstore-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}-${{ github.run_id }}-${{ matrix.platform }}-${{ matrix.shard }}
restore-keys: |
mcpp-toolstore-${{ runner.os }}-${{ matrix.toolchain }}-${{ env.MCPP_VERSION }}-
- name: Download mcpp
shell: bash
env:
MCPP_ARCHIVE: mcpp-${{ env.MCPP_VERSION }}-${{ matrix.suffix }}.${{ matrix.ext }}
MCPP_ROOT: mcpp-${{ env.MCPP_VERSION }}-${{ matrix.suffix }}
run: |
curl -L -fsS -o "$MCPP_ARCHIVE" \
"https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/${MCPP_ARCHIVE}"
case "$MCPP_ARCHIVE" in
*.zip) powershell -NoProfile -Command "Expand-Archive -Force -Path '${MCPP_ARCHIVE}' -DestinationPath '.'" ;;
*) tar -xzf "$MCPP_ARCHIVE" ;;
esac
root="$PWD/$MCPP_ROOT"
mkdir -p "$HOME/.mcpp/registry"
cp -a "$root/registry/." "$HOME/.mcpp/registry/"
# Point mcpp AT the home the caches above cover.
#
# Without this the copy above is one-way scenery: a released mcpp is
# self-contained and resolves its registry from beside its own
# executable, so every build used <tarball>/registry while the cache
# restored, and saved, ~/.mcpp/registry. Proof from the run that
# added grpc-codegen: abseil compiled from
# `<tarball>/registry/data/xpkgs/compat-x-abseil/...`, and
# xim:glibc@2.44 / xim:python@3.13.12 were downloaded again on a job
# that reported a registry cache HIT.
#
# With MCPP_HOME set, the restored registry is the one in play — so
# payloads stop being re-downloaded per job, and the tool store cache
# below lands where the build looks for it.
echo "MCPP_HOME=$HOME/.mcpp" >> "$GITHUB_ENV"
if [[ "$RUNNER_OS" == "Windows" ]]; then
echo "MCPP=$(cygpath -m "$root/${{ matrix.mcpp }}")" >> "$GITHUB_ENV"
echo "MCPP_VENDORED_XLINGS=$(cygpath -m "$root/${{ matrix.xlings }}")" >> "$GITHUB_ENV"
echo "$(cygpath -m "$root/bin")" >> "$GITHUB_PATH"
else
echo "MCPP=$root/${{ matrix.mcpp }}" >> "$GITHUB_ENV"
echo "MCPP_VENDORED_XLINGS=$root/${{ matrix.xlings }}" >> "$GITHUB_ENV"
echo "$root/bin" >> "$GITHUB_PATH"
fi
# compat.ffmpeg / compat.opencv5 carry NASM .asm sources. No host
# install and no index-refresh pre-step needed: mcpp >= 0.0.97
# resolves nasm itself through the same synchronous gate as the
# toolchain (index refresh + install + payload check BEFORE the build
# plans, mcpp#232). The sandbox copy lands in ~/.mcpp/registry, so
# the cache carries it across runs.
# ── Make the llvm leg look like a developer's box ─────────────────
# A bare GitHub runner has no libav* under /usr/include, so a
# descriptor that loses its vendored headers to the host's would still
# be green here. That is precisely how compat.ffmpeg's -idirafter hole
# survived: nothing in CI had a system ffmpeg to lose to. Installing
# the dev headers is what turns this leg into a real test of include
# ORDER rather than of include EXISTENCE.
#
# Deliberately only on the llvm leg. The default leg reaches its
# compiler through --sysroot, so it is immune by construction and
# installing there would change an existing job's environment for no
# coverage. Keeping the blast radius to the new leg means a red here
# is always about the thing this leg was added to watch.
- name: Install host ffmpeg dev headers (llvm leg only)
if: matrix.platform == 'linux' && matrix.toolchain == 'llvm'
shell: bash
run: |
sudo apt-get update -qq
sudo apt-get install -y -qq \
libavutil-dev libavcodec-dev libavformat-dev \
libavfilter-dev libavdevice-dev libswscale-dev libswresample-dev
dpkg -l | grep -E 'libav(util|codec)-dev|libswscale-dev'
# ── Select the toolchain for this leg ─────────────────────────────
# No version pin: `llvm` resolves to whatever the toolchain family's
# current release is for the PINNED MCPP_VERSION, which is the version
# that mcpp itself vouches for. Pinning a concrete llvm here would fail
# hard the day that mcpp release stops shipping it, and the point of
# this leg is the ABSENCE of a sysroot, not a particular llvm.
- name: Select toolchain
if: matrix.toolchain != 'default'
shell: bash
run: |
"$MCPP" toolchain install ${{ matrix.toolchain }}
"$MCPP" toolchain default ${{ matrix.toolchain }}
"$MCPP" toolchain list
# ── This shard's slice of the plan ────────────────────────────────
# `select` decided WHAT runs; this decides which part of it runs HERE.
# Round-robin by position, which is what spreads the expensive members:
# opencv-module / -dnn / -unifont are adjacent in the list, so `% N`
# necessarily puts them on three different runners. A single job that
# builds all three spends 45+ minutes on opencv alone.
# ── This shard's slice ────────────────────────────────────────────
# Already decided by `select` (measured-time bin packing, see
# tests/plan_shards.lua). Arrives as data, so a runner needs no lua.
- name: Take this shard's members
shell: bash
run: |
mine='${{ fromJSON(needs.select.outputs.plan)[matrix.platform][format('{0}', matrix.shard)] }}'
echo "MEMBERS=$mine" >> "$GITHUB_ENV"
echo "shard ${{ matrix.shard }}/${{ matrix.shards }}: ${mine:-<none>}"
# ── Refresh the PUBLISHED index before testing ────────────────────
# Most members resolve everything from this checkout, but a member that
# redirects a namespace other than `compat` gets the REST from the
# published index — and nothing here ever refreshed it. The snapshot in
# play is whatever the pinned mcpp release vendored (the Download step
# `cp -a`s the release's registry/ over ~/.mcpp/registry, on top of the
# restored cache), so it is by construction older than main, and it
# never moves: the cache is saved with that same stale copy inside it.
#
# mcpp does refresh on a miss for a DIRECT dependency, which is why this
# went unnoticed — the gap is a Form-A package's TRANSITIVE dependency.
# tests/examples/godot-cpp-module hit it head-on: the module package's
# own compat.godot-cpp dep resolved against a snapshot predating the
# commit that added it, and failed with `index: local index <sha> (never
# refreshed)` even though the artifact had already been republished.
# (Older members never noticed: their compat packages have been in the
# index far longer than any snapshot.)
- name: Refresh the published package index
shell: bash
env:
MCPP_INDEX_MIRROR: GLOBAL
run: |
"$MCPP" index update
- name: mcpp test (workspace or affected members)
shell: bash
env:
MCPP_INDEX_MIRROR: GLOBAL
# The GLOBAL package build cache is on (mcpp >= 2026.7.30.2), which
# is the default — this step used to set `MCPP_BUILD_CACHE: local`
# and no longer does.
#
# That bypass existed for mcpp#344: object-path disambiguation fires
# on basename collisions across the WHOLE build dir — i.e. on what
# the CONSUMER pulls in — while the cache key covered only the
# dependency, so one entry could hold two layouts and ninja died at
# graph time with "missing and no known rule to make it". #344
# landed in 2026.8.3.4 with per-package Merkle keys that cover the
# consumer-dependent layout, so the reason is gone.
#
# Keeping it cost real time, and the full run is where it showed:
# with `local`, EVERY member recompiles EVERY dependency from
# scratch. 59 members that mostly share abseil / protobuf / opencv
# meant the same sources were built over and over —
#
# linux 2h30m -> cancelled at the 150-minute timeout
# windows 2h20m
# macos 1h26m
#
# — and a workspace cannot be validated by a job that cannot finish.
# With the cache on, a given (package, version, features, toolchain)
# is built once per run and every later member hits it.
run: |
"$MCPP" --version
# No `timeout` wrapper: absent on macOS runners; job-level timeout-minutes bounds it.
# One code path: the shard step above already expanded `__ALL__`
# into this runner's actual member names, so `mcpp test --workspace`
# — which would ignore the sharding and rebuild everything here — is
# gone.
#
# tests/run_members.sh is the SAME script you run locally. A timing
# table that only exists in CI cannot be used while deciding what to
# optimise, and a local harness that differs from CI measures
# something else.
if [ -z "$MEMBERS" ]; then
echo "No workspace member affected by this change — nothing to test."
else
MCPP_TIMINGS="$PWD/timings.tsv" bash tests/run_members.sh $MEMBERS
fi
# Per-shard timings, merged by the `timings` job below. `always()`: a
# run that failed is exactly when knowing where the time went matters.
- name: Upload this shard's timings
if: always() && hashFiles('timings.tsv') != ''
uses: actions/upload-artifact@v4
with:
# The toolchain is part of the name because upload-artifact@v4
# refuses duplicates: without it both linux legs would race to
# upload `timings-linux-0` and the second one would fail the job.
name: timings-${{ matrix.platform }}-${{ matrix.toolchain }}-${{ matrix.shard }}
path: timings.tsv
retention-days: 14
# install()-driven packages (openssl, openblas) build through their own
# Make/Configure system, whose output xim's interface mode swallows; a
# failed hook surfaces only as `E_INTERNAL: [<pkg>] failed:`. Each writes
# a log into its install prefix, so on failure surface those — otherwise
# diagnosing a platform-specific build break costs a full CI round-trip
# per guess.
- name: Dump install() build logs on failure
if: failure()
shell: bash
run: |
found=0
while IFS= read -r log; do
found=1
echo "::group::$log"
tail -80 "$log"
echo "::endgroup::"
done < <(find tests/examples "$HOME/.mcpp/registry" -name 'mcpp_*_build.log' 2>/dev/null)
[ "$found" = 1 ] || echo "no install() build logs found"
# ── Where the time went ───────────────────────────────────────────────
# Sharding hides the cost: eight runners each report their own slice, and
# nobody can see which members actually dominate. This merges them into one
# ranking per platform, in the run summary, so the next optimisation starts
# from measurement instead of a guess.
#
# `always()` — a failed run is exactly when this is worth reading.
timings:
needs: [select, workspace]
if: always() && needs.select.outputs.members != ''
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
pattern: timings-*
path: timings
continue-on-error: true
- name: Rank members by wall-clock
shell: bash
run: |
shopt -s nullglob
files=(timings/*/timings.tsv)
if [ ${#files[@]} -eq 0 ]; then
echo "no timing data (every shard skipped or failed before testing)" \
>> "$GITHUB_STEP_SUMMARY"
exit 0
fi
# Artifact name carries platform AND toolchain leg:
# timings-<platform>-<toolchain>-<shard>. Ranking per LEG, not per
# platform: the two linux legs are different builds and averaging
# them would describe neither.
for leg in linux-default linux-llvm macos-default windows-default; do
rows=$(mktemp)
for f in timings/timings-$leg-*/timings.tsv; do
[ -f "$f" ] && cat "$f" >> "$rows"
done
[ -s "$rows" ] || { rm -f "$rows"; continue; }
total=$(awk -F'\t' '{s += $1} END {print s+0}' "$rows")
count=$(wc -l < "$rows")
{
echo "### $leg — ${count} member(s), ${total}s of member wall-clock"
echo
echo "| rank | seconds | share | member | result |"
echo "|---:|---:|---:|---|---|"
sort -rn "$rows" | awk -F'\t' -v tot="$total" '
{ pct = tot > 0 ? ($1 * 100 / tot) : 0
printf "| %d | %s | %.1f%% | `%s` | %s |\n", NR, $1, pct, $2, $3 }'
echo
} >> "$GITHUB_STEP_SUMMARY"
rm -f "$rows"
done
echo "_Total is the SUM across shards; wall-clock is the slowest shard._" \
>> "$GITHUB_STEP_SUMMARY"
# The table that feeds the NEXT run's sharding. Emitted as an
# artifact rather than committed automatically: a number that
# rewrites itself on every run would make every diff noisy and would
# silently absorb a one-off slow runner. Refresh it deliberately —
# download this artifact and replace tests/member-timings.tsv when
# the numbers have actually moved.
# Top 10 per platform, not every member.
#
# plan_shards prices anything absent at a fixed default, and the
# packing decisions are made by the heavy members anyway — on linux
# the top 10 of 67 are 69% of the total, and dropping the other 57
# moves the slowest shard by 0 minutes. A table that lists everyone
# goes stale the moment someone adds a test; one that lists ten only
# goes stale when the heavy set actually changes. That is the whole
# point: adding a member must not require touching this file.
{
echo "# <platform>\t<member>\t<seconds> — the HEAVY members only, top 10"
echo "# per platform, from run ${{ github.run_id }}. Everything else is"
echo "# priced at plan_shards.lua's fixed default; adding a test does NOT"
echo "# require touching this file. Refresh when a member becomes heavy"
echo "# enough to enter the top 10, or one of these numbers moves."
# ONLY the default-toolchain leg feeds this table. The llvm leg
# runs the SAME members again, so globbing every leg would put two
# rows per (platform, member) into the file — `sort -u` keeps both,
# since the seconds differ. Both legs are planned from these
# numbers, so the default leg is the right single baseline.
#
# Then: heaviest first, keep 10. `sort -u` before that so a member
# reported by two shards cannot occupy two of the ten slots.
for plat in linux macos windows; do
for f in timings/timings-$plat-default-*/timings.tsv; do
[ -f "$f" ] || continue
awk -F'\t' -v p="$plat" '{ printf "%s\t%s\t%s\n", p, $2, $1 }' "$f"
done | sort -u | sort -t"$(printf '\t')" -k3,3nr | head -10
done
} > member-timings.tsv
echo "wrote member-timings.tsv ($(grep -vc '^#' member-timings.tsv) rows, top 10/platform)"
- name: Upload the timing table for the next run's sharding
if: always() && hashFiles('member-timings.tsv') != ''
uses: actions/upload-artifact@v4
with:
name: member-timings
path: member-timings.tsv
retention-days: 90