chore: commit TTS eval harnesses + agent operational notes - #151
Open
elimelt wants to merge 2 commits into
Open
Conversation
- .notes/frontmatter.yml: optional authors field (flat list of lowercase slugs) plus conventions; absence of model entries implies human-only - AGENTS.md: agents append their model slug when creating or materially editing a note; no backfill onto untouched notes - content/templates/*: authors block with elimelt as default - scripts/validate_notes.py: reject authors when present but not a YAML list Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The eval/replay tooling used to develop the TTS reader lived only in
.quartz/, which is gitignored and wiped whenever the Quartz ref
changes. Preserve it under quartz-site/tts-reader/eval/:
- prompts.mjs prompt variants; S3 is the shipped system prompt
- tts-extract.mjs shared extraction mirroring components.js
(textOf/cleanTex/needsRewrite/collectBlocks)
- eval-semantic.mjs 18-case known-answer suite for math-to-speech
(transpose/inverse/prime/norm/binom/etc.)
- eval-perblock.mjs whole-block rewrites on real built notes:
residual LaTeX, refusal, fidelity, length ratio
- timeline.mjs full-pipeline replay against a built note with
real mp3 durations (ffprobe): TTFT, stalls,
wall-vs-audio
- inspect-inputs.mjs / inspect-fp.mjs gate/input inspection helpers
These import hast-util-from-html, which only resolves inside the
Quartz checkout, so they are copied into .quartz/ to run (commands in
docs/AGENTS.md).
docs/AGENTS.md records the operational knowledge: architecture
invariants (block extraction incl. .katex-display, urgency skip, LLM
serialization, cache versioning), backend behavior (single resident
model, concurrency collapse, prompt-cache economics, think flags),
model-selection results, prompt-design lessons, reference timeline
numbers, and known gaps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Preserves the TTS reader's eval/replay tooling and records the operational knowledge gathered while building the feature (PRs #144–#148). Everything previously lived only in
.quartz/, which is gitignored and wiped whenever the Quartz ref changes — this PR makes it durable.quartz-site/tts-reader/eval/prompts.mjsS3is the shipped system prompttts-extract.mjscomponents.js(textOf/cleanTex/needsRewrite/collectBlocks)eval-semantic.mjseval-perblock.mjstimeline.mjsinspect-inputs.mjs,inspect-fp.mjsThey import
hast-util-from-html(resolvable only inside the Quartz checkout), so the run recipe is copy-into-.quartz/— documented in the notes.docs/AGENTS.mdAgent-facing operational notes:
.katex-displaysibling-span gotcha), the rewrite gate, urgency skip, LLM serialization, cache versioning rulesthinkflags per model familyNo changes to shipped site code.