Skip to content

Experiment: Port MySQL-on-SQLite to LALR(1) parser#432

Draft
JanJakes wants to merge 11 commits into
trunkfrom
lalr-parser-driver
Draft

Experiment: Port MySQL-on-SQLite to LALR(1) parser#432
JanJakes wants to merge 11 commits into
trunkfrom
lalr-parser-driver

Conversation

@JanJakes

@JanJakes JanJakes commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

An end-to-end experiment that ports the MySQL-on-SQLite driver from its hand-written recursive parser to the LALR(1) parser merged in #429, consumed as a real Composer dependency.

All tests pass (565 in mysql-on-sqlite, 131 in mysql-parser, including the MySQL server corpus pin), and the diff is net-negative: it deletes the driver's old parser machinery.

What it does

  • Package reuse via Composer. wordpress/mysql-parser gets a classmap autoloader (WordPress file naming rules out PSR-4) and exposes WP_MySQL_Parser::PARSE_TABLE_PATH for the generated parse table. The driver requires it through a Composer path repository — a vendor symlink, so there's one source of truth and nothing is duplicated. The driver's old parser machinery (grammar, lexer, parse-tree classes, and the native Rust parser fork) is removed entirely.
  • Driver port. Statement dispatch, the query translation layer, and the information-schema builder are re-keyed to the official sql_yacc.yy rule names and tree shapes. Multi-statement input is split on top-level ; (the grammar parses one statement, the way MySQL clients do); create_parser()/next_query() becomes parse_mysql_query(). The info-schema builder is verified byte-exact against the old builder across a DDL battery (data types, constraints, indexes, table options); multi-column ADD COLUMN (a INT, b INT), which crashed the old builder, is now recorded correctly.
  • Parser refinements the port surfaced.
    • Empty reductions (opt_*, Bison mid-rule $@N) produce no AST nodes, so consumers see an optional clause only when it's present.
    • WP_Parser_Node::get_flattened_child_nodes() iterates left-recursive grammar lists (list: list ',' item) as if flat.
    • ANSI_QUOTES lexer mode plus a driver-side parse retry, for double-quoted identifiers that WordPress emits (e.g. dbDelta) but MySQL rejects without the mode.
    • Lexer fixes: six-digit /*!080400 ... */ version comments, IGNORE_SPACE token ranges, backtick-identifier unescaping, and binary-safe WP_MySQL_Token::get_value().
  • Deployment & CI. Docker environments install the driver's Composer deps and mount the package; the plugin-zip build resolves the path symlink into a pruned copy; the driver workflow also triggers on parser-package changes; the native parser extension jobs are removed (packages/php-ext-wp-mysql-parser is orphaned by this branch).

What it doesn't do yet

Testing

cd packages/mysql-on-sqlite && composer install && composer run test
cd packages/mysql-parser && composer install && composer run test
composer run build-sqlite-plugin-zip

@github-actions

github-actions Bot commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🤖 Lexer benchmark

Changes to lexer-related files were detected and triggered a benchmark:

Config Base (QPS) This PR (QPS) Speedup
no JIT 86,931 87,775 1.01×
tracing JIT 185,832 181,933 0.98×

Note: Hosted runners are noisy, and absolute numbers vary. Treat the results with caution and verify them locally.

To reproduce locally:

cd packages/mysql-on-sqlite && composer run bench-lexer

@JanJakes JanJakes force-pushed the lalr-parser-driver branch 4 times, most recently from 6ede829 to bee436b Compare June 12, 2026 09:01
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from bee436b to 076e3db Compare June 12, 2026 09:50
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 076e3db to 40a90cf Compare June 12, 2026 14:39
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 40a90cf to e75997a Compare June 12, 2026 14:56
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from e75997a to 982dd0f Compare June 12, 2026 15:28
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 982dd0f to 7730323 Compare June 12, 2026 19:09
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 7730323 to 9504ce3 Compare June 12, 2026 19:19
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 9504ce3 to d5ddec4 Compare June 12, 2026 20:36
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from d5ddec4 to b59b50b Compare June 13, 2026 13:08
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from b59b50b to 9aa6c4e Compare June 13, 2026 13:19
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 9aa6c4e to 00e9a3a Compare June 13, 2026 14:10
@JanJakes JanJakes force-pushed the lalr-parser branch 2 times, most recently from a9f8619 to aca2c38 Compare June 19, 2026 09:22
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 00e9a3a to 187593a Compare June 19, 2026 09:35
@JanJakes JanJakes mentioned this pull request Jun 19, 2026
Merged
@JanJakes JanJakes force-pushed the lalr-parser-driver branch 2 times, most recently from 816ee66 to 47a1035 Compare June 19, 2026 14:54
adamziel pushed a commit that referenced this pull request Jun 23, 2026
@JanJakes
LALR(1) parser from official MySQL grammar (#429)
49d783b 
> [!NOTE]
> The changed line numbers are misleading—about 115,000 added lines is
just a testing query corpus.
> (Copied to the new `mysql-parser` package from `mysql-on-sqlite`.)

## LALR(1) parser from official MySQL grammar

A new experimental **`packages/mysql-parser` package** that implements a
universal **LALR(1) parser** and builds a MySQL parse table from the
**official MySQL grammar**.

This is the initial implementation, not used anywhere in the driver yet.

A full driver migration to this new parser is AI-prototyped in
#432.

### What it does

- **Grammar processing pipeline:** Fetch sources → Bison → generate
parse table and token data.
- **Lexer:** The existing MySQL lexer was copied and adapted to the new
LALR(1) grammar.
- **Parser:** A new universal LALR(1) parser implementation.
- **MySQL grammar:** A compacted MySQL 8.4 LTS grammar, extracted using
the grammar processing pipeline.
- **MySQL query corpus:** The ~70k MySQL query corpus was copied and
updated to MySQL 8.4 LTS.
- **Benchmark:** A no-JIT/JIT lexer + parser benchmark.
- **Test suite:** New tests and a CI job.

### What it doesn't do yet

- **Replace the current parser:** It's a standalone package that doesn't
replace the existing parser yet.
- **Multi-version:** For now, the parser only tracks MySQL 8.4 LTS.
Multi-version will be done as a follow-up.

### Benchmarks

Measured on MacBook Pro M4 Max on PHP 8.4, the package's 8.4.10 corpus
~70k queries, end-to-end (lex + parse), best of 5 timed passes after 2
warmups:

| Metric | LL (trunk) | LALR (this) |
| --- | --- | --- |
| Throughput, no JIT | 11,010 QPS | **59,457 QPS** |
| Throughput, warm JIT | 24,393 QPS | **112,759 QPS** |
| Cold boot, no opcache | **~1.9 ms** | ~2.7 ms |
| Warm boot, opcache | ~0.6 ms | **~0.3 ms** |
| Memory, no opcache | **~3.4 MB** | ~5.4 MB |
| Memory, opcache worker | **~1.8 MB** | ~3.1 MB |
| Generated parser/table file size | **65 KB** | 177 KB |
| Full size (lexer + parser + grammar) | **246 KB** | 260 KB |

This parser is over **5× faster** without JIT and over **4.5× faster**
with JIT. Cold boot is a bit slower; warm boot is faster. The memory
footprint is a bit higher, and the overall size about 14 KB higher.

#### Recognize-only

The same lex+parse runs but building **no AST**, measuring only
recognition without AST allocation:

| Throughput | LL (trunk) | LALR (this) |
| --- | --- | --- |
| no JIT | 16,359 QPS | **95,374 QPS** |
| warm JIT | 49,940 QPS | **210,032 QPS** |

Dropping AST construction lifts both by ~1.5–2×, but the gap stays
around **~4.2–5.8×**.
Base automatically changed from lalr-parser to trunk June 23, 2026 22:50
JanJakes added 11 commits July 10, 2026 17:31
@JanJakes
Make WP_MySQL_Token::get_value() binary-safe for string literals
b8e4e7b 
The final backslash-stripping step used preg_replace() with the "u" (UTF-8)
modifier. That modifier makes PCRE validate the whole subject as UTF-8 and
return null on the first invalid byte; since get_value() is typed ": string",
the null turned into a fatal TypeError. MySQL string literals may legitimately
carry non-UTF-8 bytes (binary or other-charset payloads), and the lexer scans
them at the byte level, so reading the value of such a literal crashed.

Switch the modifier to "s" (DOTALL). A byte-wise strip is binary-safe, yields
identical results for valid UTF-8 (no continuation byte is a backslash), and
additionally handles a backslash preceding a newline byte.
@JanJakes
Stop unescaping backslashes in backtick-quoted identifier values
d732abe 
WP_MySQL_Token::get_value() routed backtick-quoted identifiers through the same
backslash-unescaping path as string literals. In MySQL a backslash is never an
escape inside `...` identifiers; only a doubled backtick is. As a result an
identifier such as `a\nb` came back as "a<newline>b", silently corrupting any
table, column, or alias name that contains a backslash.

Treat backtick identifiers like the NO_BACKSLASH_ESCAPES path: collapse only the
doubled bounding backtick and keep every other byte literal.
@JanJakes
Keep IGNORE_SPACE whitespace out of function-name token ranges
0398dfe 
When resolving a function keyword (SYM_FN), the lexer peeks for a following "("
and, under SQL_MODE_IGNORE_SPACE, skips intervening whitespace first. It skipped
by advancing bytes_already_read and never restored it. When no "(" followed, the
keyword was emitted as an IDENTIFIER whose length — derived from
bytes_already_read in produce() — now covered the trailing whitespace, so the
extracted value was e.g. "COUNT " instead of "COUNT". Under this ANSI-style mode
a column or table named after a function would resolve to the wrong identifier.

Peek with a local index instead of mutating bytes_already_read, so the token's
byte range ends at the keyword and the next scan consumes the whitespace.
@JanJakes
Support six-digit version comments in the lexer
ad31463 
read_mysql_comment() read at most five version digits, so a six-digit MMmmrr
version comment — added in MySQL 8.4 — was misparsed: /*!100000 ... */ gated as
version 10000 instead of 100000, and the sixth digit of /*!080400 ... */ leaked
into the comment body as SQL.

Mirror MySQL's own lexer rule (sql/sql_lex.cc): the first five characters must be
digits; a sixth digit immediately followed by whitespace extends the version to
six digits; otherwise the version stays five digits and any extra is content.
@JanJakes
Add a flattened child node accessor for grammar list rules
7da96fa 
Left-recursive grammar list rules nest through their own rule name
("list: list ',' item | item"). The new accessor collects child nodes
of the whole nested chain in source order, as if the list were flat,
which is how AST consumers want to iterate list items.
@JanJakes
Support the ANSI_QUOTES SQL mode in the lexer
3f9cc7b 
With the ANSI_QUOTES SQL mode, MySQL treats double-quoted text as a
quoted identifier instead of a string literal. Emit an identifier token
for it, so identifier positions accept double-quoted names.
@JanJakes
Wire the SQLite driver to the wordpress/mysql-parser package
02ade4f 
Replace the hand-written recursive parser with the table-driven LALR(1)
parser generated from MySQL's official grammar, consumed as a Composer
dependency:

- Require wordpress/mysql-parser, resolved from the monorepo sibling
  package via a Composer path repository, and load it through the
  Composer autoloader in the driver loader.
- Drop the old parser machinery (WP_Parser, WP_Parser_Grammar, the
  lexer, the parse tree classes, and mysql-grammar.php), all provided
  by the parser package now, and the native parser fork, which is bound
  to the old grammar contract.
- Parse multi-statement input by splitting the token stream on top-level
  ';' separators, as the grammar parses a single statement (this is how
  MySQL clients split multi-statement input).
- Re-key the statement dispatch to the sql_yacc.yy rule names and map
  keyword token constants to the grammar keyword table.

The translation layer still needs to be ported to the new AST shapes.
@JanJakes
Port the query translation layer to the new MySQL AST
c8d6033 
Re-key the SQL-to-SQLite translation from the old hand-written grammar
to the sql_yacc.yy rule names and tree shapes:

- Rewrite the translate() special cases and per-statement handlers
  (SELECT, INSERT/REPLACE, UPDATE, DELETE, DDL, SHOW, SET, USE,
  transactions and locking, administration statements).
- Iterate grammar lists with the flattened child node accessor, as
  lists are left-recursive in the new grammar.
- Walk JOINs recursively when building the table reference map, as
  joins nest through the left operand in the new grammar.
- Retry parsing with the ANSI_QUOTES SQL mode when a query fails to
  parse. MySQL rejects double-quoted identifiers without ANSI_QUOTES,
  but WordPress relies on them (dbDelta can produce double-quoted index
  names) and the previous parser accepted them.
@JanJakes
Port the information schema builder to the new MySQL AST
473fe1b 
Re-key CREATE TABLE, ALTER TABLE, and index statement analysis to the
sql_yacc.yy rule names and tree shapes. The recorded information schema
rows are unchanged: a battery of DDL statements covering all supported
data types, constraints, indexes, and table options produces the exact
same rows as the previous parser and builder.

Multi-column ADD COLUMN (a INT, b INT) is now recorded correctly; the
previous builder crashed on it.
@JanJakes
Adapt the test suites and tools to the mysql-parser package
2df23ac 
The lexer, parser, token data, and parse tree classes are tested in the
wordpress/mysql-parser package now:

- Remove the lexer and parser test suites from the driver package (the
  corpus data stays here; the parser package corpus test reads it from
  the sibling package and skips when it is not available).
- Move the parse tree node tests to the parser package and cover the
  new flattened child node accessor.
- Remove the native parser extension tests and tools, which are bound
  to the old grammar contract.
- Update the AST dump and benchmark tools to the new parser API.
@JanJakes
Adapt the WordPress environments, plugin build, and CI workflows
29a08df 
The SQLite driver now loads the MySQL parser as a Composer dependency,
and the native parser extension bound to the old grammar is gone:

- Install the driver Composer dependencies in the WordPress test setup
  and mount the package vendor directory and the parser package into
  the WordPress containers.
- Bundle the driver's production Composer dependencies into the plugin
  zip, resolving the path-repository symlink into a real copy of the
  parser package.
- Run the driver test workflow against changes to the parser package
  and drop the native parser extension jobs and setup scripts.
- Install the driver Composer dependencies in the lexer benchmark
  workflow.
@JanJakes JanJakes force-pushed the lalr-parser-driver branch from 47a1035 to 29a08df Compare July 10, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JanJakes