HTML API: Implement foster parenting#83
Draft
sirreal wants to merge 13 commits into
Draft
Conversation
Introduce the supporting operations that the adoption agency algorithm and active formatting element reconstruction require on the two parser stacks. On the stack of open elements: - Extract the "in scope" element list into a shared class constant. - Add `has_node_in_scope()`, which reports whether a specific node (rather than any element of a given tag name) is in scope. The adoption agency algorithm must test a specific formatting element, regardless of other open elements sharing its tag name. On the list of active formatting elements, add position-indexed operations (`position_of()`, `remove_at()`, `insert_at()`, `replace_node()`) so entries can be cloned and replaced in place as the algorithms direct. These additions are unused until the algorithms are implemented and do not change parsing behavior.
Implement the adoption agency algorithm and active formatting element reconstruction so the HTML Processor handles misnested formatting elements instead of bailing. Previously the processor stopped whenever a document required reconstructing implicitly-closed formatting elements (e.g. `<p><b>1<p>2`) or running the adoption agency algorithm (e.g. `<b>1<p>2</b>3`). Both are now supported: - `reconstruct_active_formatting_elements()` reopens the run of unclosed formatting elements at the end of the list, per the specification's rewind/advance/create steps. - `run_adoption_agency_algorithm()` implements the full algorithm, including the furthest-block case and the "any other end tag" fallback. - The "Noah's Ark clause" limits the list of active formatting elements to three equivalent entries (same tag name, namespace, and attributes). Because the processor visits a document in a single pass, it cannot relocate nodes it has already reported. The parser's state (the stack of open elements and the list of active formatting elements) is maintained exactly as the specification requires, so every token visited after these algorithms run is reported with the ancestor chain a browser would produce. Nodes which were already visited when a misnesting is discovered remain where they were found. Formatting elements reopened by the parser are reported as "virtual" nodes. Reading an attribute, class, or qualified name of such a node reports the value from the tag which opened the original element; these nodes cannot be modified. Supporting this required hardening stack-event provenance so a single source tag never produces two visitor events: pushes are matched to the current token by identity, and each tag closer is matched to at most one popped node. The html5lib test cases whose constructed trees differ only because the adoption agency algorithm re-parents already-visited nodes are skip-listed with a shared reason; each was verified to match browser behavior for parser state and normalization. The absorbed `wpHtmlSupportRequiredActiveFormatReconstruction` test and the previous bail-asserting cases are replaced with tests of the new behavior.
A FORM end tag encountered while other elements remain open no longer stops the parser. The form element is removed from the stack of open elements using the same reconciliation the adoption agency algorithm uses, so any elements that remain open after it are reported with correct breadcrumbs. The scope check now tests the specific form element pointer rather than any FORM element in scope, matching the specification. One html5lib case (`<form><div></form><div>`) exercises a shape a single-pass token stream cannot represent: browsers keep the closed FORM as a DOM ancestor of its still-open descendants. This parser reports following content outside the closed FORM, mirroring the stack of open elements; the case is skip-listed with that reason.
# Conflicts: # tests/phpunit/tests/html-api/wpHtmlProcessorWebPlatformTests.php
The adoption agency branch's skip list was computed against the html5lib-tests data files. Trunk migrated to the Web Platform Tests tree-construction suite, in which some data files gained tests or shifted lines: - webkit01 skips shifted by two lines (0571/0586/0603 -> 0569/0584/0601). - adoption02/line0021 is a new adoption-agency reparenting case. - tests1/line0355 and tests1/line1532 previously bailed (reconstruction unsupported); with reconstruction implemented they parse but follow the old SELECT content model, dropping the B element, matching the existing customizable-SELECT skip class. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
Content found inside a TABLE element where it isn't allowed belongs at a location in the document before that table: browsers relocate it in a process called foster parenting. Until now the HTML Processor bailed whenever fostering was required, refusing to process any document with mis-nested table content. Support foster parenting in a streaming-compatible way: every node is still visited at the place it was found in the input HTML, in a single pass with O(1) memory overhead, and no buffering of table contents. A foster-parented node is marked as such when it's inserted, and its breadcrumbs report its document ancestry, which bypasses the enclosing table context: the bypassed segment of the stack of open elements is set aside while the fostered node is open and restored when it closes. The trade-off of this design is that a foster-parented node is visited where its tag or text appears in the input HTML, which is after the TABLE element it precedes in the document. Comments are never fostered and always remain in place; document ancestry is always exact; only the visitation order of fostered content relative to its table diverges from tree order. The new WP_HTML_Processor::is_foster_parented() method reports when the current node is such a node so that callers building trees can place it exactly. Details of the change: - The foster parenting flag from the specification is enabled around the 'in table' insertion mode's character and "anything else" rules, which process tokens using the 'in body' rules. It's cleared upon entering step() so that handlers which ignore a token or reprocess it in another insertion mode cannot leak the flag onto other tokens. - Insertions targeting a TABLE, TBODY, TFOOT, THEAD, or TR element while the flag is enabled mark the inserted token as foster-parented; this includes elements reconstructed from the list of active formatting elements and elements inserted by the adoption agency algorithm's placement of the "last node" when its override target is such an element. - The stack push handler records how many elements of the stack the fostered node's ancestry bypasses: everything above and including the nearest TABLE below it, or everything above the nearest TEMPLATE when fostered content belongs inside template contents. The count rides on the stack event because the stack may change before the event is visited. - Whitespace-only text in a table context looks ahead within its run of text: when non-whitespace follows in the same run, a browser fosters the entire run, including the leading whitespace. - The web-platform-tests harness now builds a realized document tree, placing foster-parented nodes as a browser would and merging adjacent text, so the tree-construction fixtures verify fostered placement exactly; 90 previously-skipped fixtures now run, with 78 passing and 12 skipped for previously-cataloged reasons: the adoption agency algorithm cannot relocate visited nodes in a single-pass parser, an A element removed from the stack of open elements while its descendants remain open cannot be held open, and the SELECT content model changes are unimplemented. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
Cover the public behavior of foster parenting support: fostered ancestry in breadcrumbs and depth, the is_foster_parented() accessor, comments remaining inside tables, the pending-table-character-tokens whitespace rules, fostering into template contents and before the nearest enclosing table, reconstruction and adoption placements, breadcrumb queries, attribute updates on fostered elements, seeking across fostered content, and normalization idempotence. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
The algorithms which clear the stack of open elements back to a table, table body, or table row context, and the algorithm which closes a table cell, stop at elements by tag name: TABLE, TEMPLATE, TBODY, TR, TD, etc. These steps in the specification refer to HTML elements, but the checks matched foreign elements with the same tag names, such as an SVG TEMPLATE, wrongly terminating the clearing and corrupting the parse. Such foreign elements were unreachable in raw table contexts while foster parenting bailed: foreign content only appeared inside cells and captions, where the mistaken matches went unobserved because surrounding algorithms cleaned up after them. Foster parenting places foreign content directly in table contexts — e.g. a fostered SVG in row context whose TEMPLATE child terminated the clearing for a TR end tag — where no later step heals the stack. Found by fuzzing against Dom\HTMLDocument: with this fix, a 20,000-case generative sweep over table-content soups reports identical ancestry for every sentinel and no structural round-trip differences between parsing an input and parsing its normalization. Also document that normalization serializes foster-parented content where its syntax was found, and add a caveat about whitespace which was separated from fostered text only by ignored syntax. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
The HTML Processor guarantees that it visits nodes in document order: before foster parenting support, it aborted whenever content belonged at a document location before a place it had already visited. Foster parenting support silently traded that guarantee away: fostered nodes are visited where they were found in the input HTML, after the TABLE element they precede in the document, so the sequence of visited nodes is no longer a pre-order traversal of the document. Existing code which walks documents relying on the order and depth of visited nodes — such as finding the end of an element's subtree by watching the depth — could be silently confused by fostered content. Preserve the document-order guarantee by default: the processor aborts on content requiring foster parenting, as it always has, unless the new WP_HTML_Processor::enable_foster_parenting() method is called before scanning begins. Enabling it is an informed trade: mis-nested table content of any size parses in a single pass, every node still reports exact document ancestry, and is_foster_parented() identifies the nodes which were visited out of document order. Because normalization cannot enable the support on its internal processor, normalize() refuses fostering documents as before; calling serialize() on a processor with foster parenting enabled serializes fostered content where its syntax was found, and the output parses into the same document. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
Specify how the HTML Processor presents foster-parented content in a streaming, single-pass parser without ever violating its document-order contract by default: - Document-order mode (default): table contents are deferred in a bounded, document-order buffer (a "table window") until the table closes; fostered content is inserted at its document position within the buffer, so flushing presents everything in document order. On overflow the buffer flushes and the parse continues exactly as before this feature existed, aborting only if fostering is required after the flush. No document parses worse than before; documents whose fostering resolves within the bound gain exact document-order presentation. - Source-order mode (opt-in): every node presents at its source position with O(1) overhead and no bound, marked via is_foster_parented(), for order-independent consumers. The specification covers routing rules for nested tables, template contents, and fostered runs; overflow semantics; visitation-index ordering for seek(); serialization in both modes; and the bound's rationale. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
enable_foster_parenting() described the method when it was the only way to process mis-nested table content. With document-order support specified as the default, the method's meaning is the mode it selects: source-order presentation, unbounded and O(1), at the cost of visiting fostered nodes after the TABLE they precede in the document. Name it enable_source_order_foster_parenting() accordingly. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
Implement the document-order presentation mode specified in foster-parenting.md: the processor now supports foster parenting by default while preserving its guarantee that nodes are visited in document order — a foster-parented node is visited before the TABLE element it precedes in the document. The contents of each outermost TABLE element are deferred in a table window until the table closes: fostered events are inserted at their document position within the window (before their anchor TABLE's push, or after the pop of the host child of the TEMPLATE contents receiving them), events inside an open fostered run follow their run, and everything else appends. Flushing the window front-to-back therefore presents its subtree in document order by construction. Content fostered into TEMPLATE contents defers even without a window, keyed to its host child's pop. Deferral is bounded by MAX_BUFFERED_TABLE_EVENTS. Ordinary table content exceeding the bound flushes the window and streams on undeferred — well-formed tables of any size parse exactly as before — and only mis-nested content discovered beyond the bound aborts the parse, as all mis-nested table content did before this change. No document parses worse than before; documents whose fostering resolves within the bound now parse in exact document order, and normalize() relocates their fostered content to its document position, producing output which needs no foster parenting when re-parsed. Because deferred tokens are visited after the lexer moved past their syntax, visiting one repositions the lexer to the token's span — pop events record their closing tag's token for this — so reads, in-place modifications, and bookmarks behave identically to undeferred visits; the lexer returns to the parse frontier before parsing continues, and internal repositioning neither counts against the caller's seek() budget nor consumes queued events. seek() now orders positions by visitation index rather than byte offset, since deferral makes the two disagree. The source-order mode behind enable_source_order_foster_parenting() is unchanged: no deferral, no bound, fostered nodes visited at their source position. The web-platform-tests harness now verifies every tree-construction fixture in both modes. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
Routing stack events can refuse to proceed when foster-parented content exceeds the deferral buffer. Two call sites popped elements outside of any handler for the exception which bail() throws: the end-of-document unwinding of unclosed elements, and the closing of a void-like node at the start of step(). An oversized fostered run reaching either path escaped as an uncaught exception; both now report through get_last_error() like every other abort. Seeking backwards clears deferred-presentation state before clearing the stacks, whose discarded teardown events could otherwise route through open table windows or fostered runs and spuriously abort a legitimate seek. Found by fuzzing with a reduced deferral bound; a regression test covers the unwinding path. Claude-Session: https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG
Owner
Author
|
Review finding:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements foster parenting in the HTML Processor: content mis-nested inside a TABLE —
<table>lost<td>found— no longer aborts the parse. Arbitrary tables can now be parsed and processed.Builds on the adoption agency + active format reconstruction branch, whose commits are included here (fostering's "anything else" path processes tokens with the in-body rules, which need reconstruction and the adoption agency).
A specification for the behavior lives in-tree at
src/wp-includes/html-api/foster-parenting.mdand is the contract for the implementation.The problem
Foster parenting relocates content which appears in the input after the
<table>tag to a document location before the table element. A streaming parser must either present such nodes out of document order — breaking the processor's guarantee that visitation is a pre-order traversal of the document, which consumers rely on for order/depth reasoning — or withhold table contents until it knows nothing more will be fostered, which is only knowable at the table's end.Two presentation modes
Document order (default). The contents of each outermost TABLE are deferred in a bounded "table window" until the table closes; fostered events are inserted at their document position within the window (before their anchor table's push, or after the host child's pop for content fostered into TEMPLATE contents), so flushing the window front-to-back presents everything in document order by construction. Consequences of the bound (
MAX_BUFFERED_TABLE_EVENTS, 1,000 events):normalize()now relocates fostered content to its document position: the output needs no foster parenting when re-parsed.Source order (opt-in).
enable_source_order_foster_parenting()removes the deferral and the bound: every node is visited at its source position with O(1) overhead, and a fostered node is visited after the TABLE it precedes in the document. Breadcrumbs report exact document ancestry in both modes; the newis_foster_parented()marks the relocated nodes. This mode suits order-independent processing of arbitrary input (per-node reads and in-place mutations keyed on tag names, attributes, or breadcrumbs).Deferred tokens are visited after the lexer has moved past their syntax, so visiting one repositions the lexer to the token's span: reads, in-place modifications, and bookmarks behave identically to undeferred visits.
seek()orders positions by visitation index rather than byte offset, since deferral makes the two disagree.Also fixed along the way: the clear-to-table-context algorithms and
close_cell()matched TEMPLATE/TD/etc. by tag name without checking the namespace; fostered foreign content (an SVGtemplatein row context) made the mismatch reachable and corrupted the parse.Verification
Dom\HTMLDocument(PHP 8.4+ oracle), 20k+ random table soups:serialize()output re-parses to the identical oracle tree (modulo the pre-existing single-pass divergence classes: adoption relocating already-visited nodes, mid-stack FORM/A removal, and the old SELECT content model).Known cost:
next_token()on a table opener may do up to a bound's worth of parsing before returning, and deferred table content is lexed twice (once parsing, once at visitation) — bounded by the same constant, not measurable in the benchmarks above.Trac ticket: none yet — unreleased branch work; a ticket will accompany upstreaming (test
@ticket TBDannotations to be updated then).Use of AI Tools
AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Fable 5
Used for: Specification, implementation, tests, and fuzz/benchmark verification, developed interactively with design direction, constraint-setting, and review by @sirreal.
https://claude.ai/code/session_01Y6FLksvN4P1b3gZFzX41DG