LLM friendly website content

[mikaalnaik]

No description provided.

[mikaalnaik]

Code review — LLM-friendly website content

What it does: Serves clean markdown representations of memos/posts/builders to LLM agents. Content is reachable two ways — an explicit .md suffix or an Accept: text/markdown header — both rewritten in proxy.ts to an internal /md/[...path] route. Adds an /llms.txt index, Content-Signal headers, and alternates.types link metadata on the HTML pages. HTML→markdown conversion runs through a unified/rehype-remark pipeline.

Well-structured and carefully reasoned. tsc --noEmit is clean. The layering (content builders → document assembly → html-to-markdown) is clean, and the inline comments explaining the non-obvious decisions (Vary/Accept caching, canonical + noindex, Docker vs public/_headers) are genuinely useful.

Strengths

• Draft leak is provably prevented. The /md route never calls setAccessToken, and apiFetch only attaches a bearer token when one is set in the request-scoped auth-token store — so the markdown route only ever fetches published content. The comment claim checks out against client.ts / auth-token.ts.
• Safe YAML frontmatter via JSON.stringify on every value — titles with : or quotes can't break the block.
• Graceful degradation in llms.txt (.catch(() => []) per source) so an upstream outage degrades to partial output rather than a 500.
• Good SEO hygiene: X-Robots-Tag: noindex + Link: rel=canonical keeps the .md variants out of the index while pointing to the canonical HTML.
• Proxy refactor correctly scopes renewSession to /memos/* + /api/auth/me, so the newly-matched /posts/* and /builders/* don't pay for session renewal.

Suggestions (all minor, none blocking)

• llms.txt note fields can break list formatting. keyMessage1 / quote are interpolated raw into - [title](url): note. If either contains a newline the list item breaks. Consider collapsing whitespace, e.g. note?.replace(/\s+/g, " ").trim().
• /md/[...path] re-runs the unified transform on every request. The upstream fetch is ISR-cached, but the HTML→markdown conversion is not (no revalidate export on the route). For hot content, adding export const revalidate = 300 would cache the rendered output too.
• rehypeAbsoluteUrls covers href/src but not srcset. Images using srcset would keep relative URLs. Likely fine for York Factory content, but worth a note.
• Content-Signal is now defined in two places (next.config.ts and public/_headers) — intentional per the comment (Docker vs Cloudflare Pages), but the two values must be kept in sync manually; a pointer comment in public/_headers back to next.config.ts would help future editors.
• /md/{type}/{slug} is directly reachable despite the "not linked directly" comment. Not a problem (it's just published markdown), just flagging it's a real public surface.

Overall: LGTM. 👍