Improve docs UX for AI agents (llms.txt, on-domain markdown, robots, meta)

[juan-malbeclabs]

Why

The docs site is solid for humans but didn't meet current conventions for consumption by AI agents / LLMs. Analysis of the built site/ surfaced concrete gaps (no llms.txt, no on-domain raw Markdown, broken "View Markdown" for translated pages, zero <meta name="description">, no robots.txt, a duplicated toolbar).

What changed

Area	Change
llms.txt	New /llms.txt (curated, sectioned index with descriptions) and /llms-full.txt (full concatenated content), generated from the default (English) locale.
On-domain Markdown	Every page now also serves clean Markdown at <page>/index.md (all 8 languages), so agents can fetch source Markdown without GitHub.
Toolbar fix	"View Markdown" / "Copy Page" / "Ask in ChatGPT/Claude" now use the on-domain .md. The old GitHub-raw link was broken for all 7 translated locales (/es/setup/ → requested docs/es/setup.md, a 404; real source is docs/setup.es.md). Removed the duplicate page-toolbar.js include that rendered the toolbar twice.
robots.txt	New docs/robots.txt explicitly allowing AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, …) and referencing the sitemap.
Meta descriptions	Added site_description + per-page description front matter. Every page now has a <meta name="description"> (was 0).
Housekeeping	Fixed faviron → favicon typo; removed get-pip.py (2 MB), hold from setup, .DS_Store; ignore .DS_Store / __pycache__.

Implementation note

The mkdocs-llmstxt plugin was the obvious choice but is incompatible with mkdocs-static-i18n — it can't resolve localized page URIs and skips every page. So llms.txt / llms-full.txt and the per-page Markdown are produced by a single MkDocs hook (hooks/emit_markdown.py) that filters to the default locale. No new CI dependency.

Verification

Built locally with mkdocs build:

• /llms.txt, /llms-full.txt, /robots.txt present at site root.
• site/setup/index.md and site/es/setup/index.md exist with clean (front-matter-stripped) Markdown, correctly translated.
• 208 pages now carry <meta name="description"> (pages without their own description fall back to site_description).
• page-toolbar.js included exactly once per page (was twice).
• Served via mkdocs serve: toolbar appears once; "View Markdown" opens the on-domain .md and works on translated pages.

🤖 Generated with Claude Code

[armcconnell]

Reviewed and built locally — all claims verified (llms.txt/llms-full.txt/robots.txt/sitemap at root, 209/209 pages with meta descriptions, on-domain .md translated and front-matter-stripped, toolbar included once, cruft removed).

Pushed a follow-up commit adding description: front matter to the remaining nav pages, so each now gets a unique <meta name="description"> and a description in llms.txt instead of the generic site_description.

Filed two non-blocking follow-ups:

• Raw on-domain .md uses source-relative links that 404 when fetched #177 — raw on-domain .md keeps source-relative links that 404 when fetched
• llms.txt SECTIONS list is hand-maintained in parallel with nav #178 — llms.txt SECTIONS list is hand-maintained in parallel with nav

Also noted (out of scope): site/ is now in .gitignore but the built output is still tracked in the index (253 files) — a git rm -r --cached site/ would finish that cleanup.

[armcconnell]

Approving. Changes do what they claim and directly improve agent readiness; verified with a local build. Per-page descriptions added in a follow-up commit; remaining items tracked in #177 and #178.