Stop superseded term pages from aborting the build - #421
Merged
Conversation
Term pages generated at version 6 and earlier call Docsy's alert shortcode. vfb-nova does not ship it, and a missing shortcode template is a hard build error, so a single superseded file anywhere in ~763k pages aborts the run -- twice this week, at 93 and 38 minutes each. Same signature and rendering as Docsy's, plus .alert rules in compat.css using the theme's tokens so they are styled rather than bare. pageinfo is included pre-emptively: it is the other Docsy shortcode those pages are likely to use, and finding out the slow way costs another build.
process_term() removed only version-1 after writing a page, so a term absent from a single run's ID list kept its older file forever. The corpus still held _v6 pages with version at 9, in blog/ontologies/* and blog/releases/datasets alike, including one named _v6.md from a term with an empty short_form. Orphans are live pages: they are walked and parsed on every build, they can collide on URL with the current version, and ones predating the move off Docsy call shortcodes the theme no longer ships -- which aborts the build outright. Prunes once per group with a single directory scan instead of a stat per term per version. The per-term form would cost millions of extra syscalls across ~763k terms, and this corpus lives on an NFS volume serving a few hundred metadata ops per second. Net effect is one fewer stat per term than before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Two builds this week died after 93 and 38 minutes on the same class of error:
Both are superseded term pages - generator version 6, while
versionis 9 -that were never cleaned up. They predate the move off Docsy, so they call
Docsy's
alertshortcode, and a missing shortcode template is a hard error, nota warning. One orphan anywhere in ~763k pages aborts the whole run.
Why the orphans exist
process_term()removed only one generation back:A term missing from a single run's ID list keeps its older file permanently -
_v6survives the bump to 7, 8 and 9. They accumulate inblog/ontologies/*and
blog/releases/datasetsalike, including one literally named_v6.md, froma term with an empty
short_form.Orphans are not inert. They are walked and parsed on every build, they can
collide on URL with the current version of the same term, and old enough ones
break the build.
Two commits
Reimplement
alertandpageinfowith Docsy's signature and rendering, plus.alertrules incompat.cssusing the theme's own tokens so they are styledrather than bare.
pageinfois pre-emptive: it is the other Docsy shortcodethose pages likely use, and finding out the slow way costs another build.
Prune every superseded version, once per group, with a single directory
scan:
Not a stat per term per version - that would be millions of extra syscalls
across ~763k terms, and this corpus lives on an NFS volume serving a few hundred
metadata operations per second. Net effect is one fewer stat per term than
before, since the old per-term cleanup is removed.
Verified against the real cases:
A_v6.md,B_v1.mdand_v6.mdremoved;*_v9.mdand_index.mduntouched.Note on sequencing
The existing orphans have already been swept by hand. This PR stops them coming
back, and keeps a single missed file from costing a five-hour build.
What this does not fix
The
vfb/directory still holds ~650k files in one directory, which is theunderlying cause of the build's cost. Measured on the NAS:
go/at 14k entriesenumerates at 20,191/s,
vfb/at 650k collapses to 5,076/s and 253 stat/s,with 128 MiB of page cache available. Sharding by ID prefix is the real fix and
needs its own PR.