feat(download-ref): store arXiv LaTeX sources as KB full text

[chenpeizhi]

Motivation

The knowledge base's rendered .md files are the agent-facing text: advisor subagents, survey-writer, and paper-reviewer all Read/Grep them, and INDEX.md/references.bib are derived from them. Today those bodies come from PDF-to-Markdown conversion (pymupdf4llm), which is lossiest exactly where physics/math papers carry their content — equations, operators, sub/superscripts come out garbled, and two-column layouts shred sentence flow.

For arXiv papers the author's LaTeX source is available and is ground truth for the math and structure. LLM agents read raw LaTeX natively, so no conversion step is needed — the fidelity is free. In practice most KB refs are arXiv papers or journal articles with arXiv preprints, so following externalIds.ArXiv for DOI entries extends the benefit to journal-DOI bibs without giving up journal-grade metadata.

Deliberate scoping decisions (recorded in the spec):

• The .md wrapper stays canonical. The file-type contract (frontmatter, <id>_<slug>.md naming, INDEX.md regeneration, downstream skill globs, and the vendored-helper family in quantum.harness/init-harness) is untouched — only the body's source changes, marked by the new full_text: latex value.
• No pandoc tex-to-Markdown conversion — it silently mangles macro-heavy papers, and the consumer is an LLM, not a Markdown renderer.
• papis integration was evaluated and deferred — its benefits (query CLI, papis-ask RAG, Zotero interop) are complementary rather than a better version of this pipeline, and swapping the .raw/ storage backend would diverge the layout shared with the harness family. See the spec's Non-goals.

Summary

Store arXiv e-print LaTeX sources as the full-text body of rendered knowledge-base entries, instead of PDF-to-Markdown conversion. PDF rendering (pymupdf4llm) remains the fallback for refs without source.

Design spec: docs/specs/2026-08-03-arxiv-latex-source-design.md (included in this PR).

What changed

• New skills/download-ref/helpers/tex_source.py (stdlib-only): e-print download, payload detection (tarball / single-file / PDF-only submissions / withdrawn-paper HTML), path-traversal-safe tar extraction, main-file discovery, \input/\include flattening (latexpand when installed, built-in Python inliner otherwise), figure copying into .figures/, and the fetch_arxiv_source orchestrator (statuses ok/cached/pdf-only/miss; all failures degrade, one bad paper never aborts a batch).
• fetch_metadata.py: new --download-arxiv-source flag. Fetches source for arXiv manifest entries and for DOI entries whose Semantic Scholar record names an arXiv preprint (externalIds.ArXiv) — mirroring the existing arXiv-PDF paywall fallback. When the same paper appears both as an arXiv entry and as a DOI preprint, the already-fetched source is reused (tex + figures copied) instead of downloaded twice.
• render.py: render_arxiv and render_doi prefer the flattened .tex (full_text: latex, raw LaTeX under ## Full Text (LaTeX source)); PDF path unchanged otherwise.
• index.py: full_text: latex counts as full text.
• Docs: skills/download-ref/SKILL.md (preflight, Step 4/5, owns-list, checklist, common mistakes) and CLAUDE.md.

The KB contract is unchanged: same <id>_<slug>.md filenames, frontmatter keys, INDEX.md regeneration, references.bib flow, and .raw//.figures/ layout — full_text gains the value latex additively, so existing KBs and the quantum.harness/init-harness vendored-helper family are unaffected (their md_to_bibtex.py should learn latex is truthy on their next helper sync).

Testing

• 27 new unit tests (tests/test_tex_source.py, tests/test_render_tex.py, tests/test_index.py): real tar/gzip payloads, network mocked only at urlopen; covers traversal, broken symlinks, permission errors, PDF-only submissions, withdrawn papers, flatten crashes, destination overrides, and both render paths.
• Full suite: 144 passed; the 7 pre-existing failures on main (missing skills/ideas/SKILL.md, unrelated) are unchanged.
• Live end-to-end: arXiv:1806.08734 (tarball with figures) and DOI 10.22331/q-2018-08-06-79 → arXiv:1801.00862 (DOI-preprint path) — fetch, flatten, render (full_text: latex), INDEX, and idempotent (cached) re-runs verified against the real endpoints.

Review follow-ups

The three cosmetic items flagged in review (arXiv-centric module docstring, arXiv/DOI double fetch for papers listed both ways, done-checklist wording) are addressed in the final commit.

🤖 Generated with Claude Code

[chenpeizhi]

@GiggleLiu Any comments?