Skip to content

Stop superseded term pages from aborting the build - #421

Merged
Robbie1977 merged 2 commits into
masterfrom
fix/legacy-shortcodes-and-prune
Aug 14, 2026
Merged

Stop superseded term pages from aborting the build#421
Robbie1977 merged 2 commits into
masterfrom
fix/legacy-shortcodes-and-prune

Conversation

@Robbie1977

@Robbie1977 Robbie1977 commented Aug 14, 2026

Copy link
Copy Markdown

The problem

Two builds this week died after 93 and 38 minutes on the same class of error:

assemble: ".../ontologies/bfo/BFO_0000001_v6.md:16:1":
  failed to extract shortcode: template for shortcode "alert" not found
assemble: ".../releases/datasets/_v6.md:25:1":
  failed to extract shortcode: template for shortcode "alert" not found

Both are superseded term pages - generator version 6, while version is 9 -
that were never cleaned up. They predate the move off Docsy, so they call
Docsy's alert shortcode, and a missing shortcode template is a hard error, not
a warning. One orphan anywhere in ~763k pages aborts the whole run.

Why the orphans exist

process_term() removed only one generation back:

old_filename = term_id + "_v" + str(version - 1) + ".md"

A term missing from a single run's ID list keeps its older file permanently -
_v6 survives the bump to 7, 8 and 9. They accumulate in blog/ontologies/*
and blog/releases/datasets alike, including one literally named _v6.md, from
a term with an empty short_form.

Orphans are not inert. They are walked and parsed on every build, they can
collide on URL with the current version of the same term, and old enough ones
break the build.

Two commits

Reimplement alert and pageinfo with Docsy's signature and rendering, plus
.alert rules in compat.css using the theme's own tokens so they are styled
rather than bare. pageinfo is pre-emptive: it is the other Docsy shortcode
those pages likely use, and finding out the slow way costs another build.

Prune every superseded version, once per group, with a single directory
scan:

with os.scandir(".") as entries:
    for entry in entries:
        match = _VERSION_SUFFIX.search(entry.name)
        if match and int(match.group(1)) < version:
            os.remove(entry.name)

Not a stat per term per version - that would be millions of extra syscalls
across ~763k terms, and this corpus lives on an NFS volume serving a few hundred
metadata operations per second. Net effect is one fewer stat per term than
before
, since the old per-term cleanup is removed.

Verified against the real cases: A_v6.md, B_v1.md and _v6.md removed;
*_v9.md and _index.md untouched.

Note on sequencing

The existing orphans have already been swept by hand. This PR stops them coming
back, and keeps a single missed file from costing a five-hour build.

What this does not fix

The vfb/ directory still holds ~650k files in one directory, which is the
underlying cause of the build's cost. Measured on the NAS: go/ at 14k entries
enumerates at 20,191/s, vfb/ at 650k collapses to 5,076/s and 253 stat/s,
with 128 MiB of page cache available. Sharding by ID prefix is the real fix and
needs its own PR.

Term pages generated at version 6 and earlier call Docsy's alert shortcode.
vfb-nova does not ship it, and a missing shortcode template is a hard build
error, so a single superseded file anywhere in ~763k pages aborts the run --
twice this week, at 93 and 38 minutes each.

Same signature and rendering as Docsy's, plus .alert rules in compat.css using
the theme's tokens so they are styled rather than bare. pageinfo is included
pre-emptively: it is the other Docsy shortcode those pages are likely to use,
and finding out the slow way costs another build.
process_term() removed only version-1 after writing a page, so a term absent
from a single run's ID list kept its older file forever. The corpus still held
_v6 pages with version at 9, in blog/ontologies/* and blog/releases/datasets
alike, including one named _v6.md from a term with an empty short_form.

Orphans are live pages: they are walked and parsed on every build, they can
collide on URL with the current version, and ones predating the move off Docsy
call shortcodes the theme no longer ships -- which aborts the build outright.

Prunes once per group with a single directory scan instead of a stat per term
per version. The per-term form would cost millions of extra syscalls across
~763k terms, and this corpus lives on an NFS volume serving a few hundred
metadata ops per second. Net effect is one fewer stat per term than before.
@Robbie1977
Robbie1977 merged commit 91479c7 into master Aug 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant