Add Zotero private-library cache workflows

docs/how-to/export-cache-to-zotero.md

@@ -0,0 +1,114 @@
+# Export a Reference Cache to Zotero
+
+This workflow turns a checked-in public reference cache into a Zotero collection,
+asks Zotero to find available PDFs, and then inventories the resulting attachments
+without putting closed manuscripts into the project repository.
+
+The three stores remain separate:
+
+| Store | Purpose | Safe to commit? |
+|---|---|---:|
+| Project `references_cache` | Reproducible public validation evidence | Yes |
+| Zotero library | User-managed references and PDF attachments | No |
+| Private research cache | Extracted user-library text for agent context | No |
+
+Normal validation reads only the first store.
+
+## 1. Export metadata from the public cache
+
+Run the exporter from the `linkml-reference-validator` checkout. For dismech:
+
+```bash
+uv run linkml-reference-validator cache export \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --needs-full-text \
+  --output /Users/cjm/Downloads/dismech-zotero.json
+```
+
+`--needs-full-text` is the default. It excludes cache entries that already have
+full text, excludes non-publication identifiers, and deduplicates by normalized
+DOI and then PMID. The CSL JSON output uses an explicit metadata allowlist:
+
+- title and authors;
+- journal and year;
+- DOI; or
+- PMID and its public PubMed URL.
+
+It never contains cached article text, excerpts, provenance, PDFs, local paths,
+or private-cache data. The command refuses to replace an existing output file;
+use `--force` only after checking the destination. Use `--all` if you explicitly
+want eligible references that already contain full text.
+
+## 2. Import the file into Zotero
+
+In Zotero:
+
+1. Choose **File → Import**.
+2. Choose **A file**.
+3. Select `dismech-zotero.json`.
+4. Keep the imported records in a project-specific collection such as
+   **dismech**.
+
+CSL JSON is a standard format supported by Zotero. Importing the file adds only
+bibliographic records; it does not copy anything from the validator's caches.
+
+## 3. Ask Zotero to find PDFs
+
+Select one or more top-level journal articles in the center item list,
+right-click the selection, and choose **Find Full Text**. It is a built-in Zotero
+command, not a plugin.
+
+![Zotero Find Full Text processing a batch of imported references](../assets/zotero-find-full-text.png)
+
+The command appears for an eligible parent item that has a DOI and does not
+already have a file attachment. It may also be unavailable in a group library
+where files cannot be added. If it is absent, first test a single article in
+your personal library and confirm that its DOI field is populated.
+
+For a large import, start with a modest batch rather than all records at once.
+Publishers can temporarily rate-limit repeated requests, and a later retry may
+find additional files. Zotero reports **Full Text PDF** or **No file found** for
+each attempted item.
+
+## 4. Inventory the enriched Zotero library
+
+Once Zotero finishes, scan the project cache again:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --dry-run
+```
+
+This read-only scan matches exact DOI, PMID, or PMCID identifiers and reports
+which cached references now have usable Zotero text or PDF attachments. It does
+not invoke **Find Full Text** and does not modify Zotero.
+
+After reviewing the matches, materialize them into the separate private research
+cache:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --apply
+```
+
+Apply mode leaves `references_cache` unchanged. Private content goes to
+`~/.cache/linkml-reference-validator/private` by default with owner-only
+permissions. Agents may inspect it for context, but validation cannot use it as
+evidence.
+
+## 5. Validate reproducibly
+
+Run validation normally. Results remain reproducible because the validator
+reads the public project cache and accepts only public full-text provider results.
+Neither the contents of Zotero nor the machine-specific private cache affect
+validation.
+
+## See also
+
+- [Fetching Full Text and PDFs](fetch-full-text-and-pdfs.md)
+- [Zotero: importing standardized formats](https://www.zotero.org/support/kb/importing_standardized_formats)
+- [Zotero: adding files to a library](https://www.zotero.org/support/attaching_files)

docs/how-to/fetch-full-text-and-pdfs.md

@@ -46,9 +46,11 @@ your config YAML or on the `ReferenceValidationConfig` object):
 |-----|---------|-------------|
 | `fetch_full_text` | `true` | Attempt to obtain full text via the provider chain when a metadata source does not already return full text. |
 | `full_text_providers` | `[pmc, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. |
+| `private_cache_dir` | `~/.cache/linkml-reference-validator/private` | Separate research cache for closed/user-library full text; never read by validation. |
 | `pdf_backend` | `pypdf` | Name of the PDF text-extraction backend. |
 | `download_pdfs` | `true` | If true, persist downloaded PDFs to the files cache directory. |
 | `full_text_providers_file` | `null` | Optional path to a YAML file defining custom full-text providers. |
+| `zotero_base_url` | `http://localhost:23119/api/users/0` | Read-only Zotero local API library used by the opt-in `zotero` provider. |
 
 Two existing keys are also reused by the full-text machinery:
 
@@ -71,6 +73,106 @@ pdf_backend: pypdf
 download_pdfs: true
 ```
 
+## Private manuscripts in Zotero
+
+The opt-in **`zotero`** provider can find non-open manuscripts that are already
+in your personal Zotero library. It matches exact DOI, PMID, or PMCID identifiers;
+it does not automatically accept fuzzy title matches.
+
+Zotero must be running, and its local API must be enabled in Zotero's advanced
+settings. Inventory the reference cache before changing it:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --dry-run
+```
+
+The report distinguishes `found`, `not_found`, `already_full_text`, and `error`.
+Dry-run is the default and never changes cache files. After reviewing exact
+matches, apply usable text with:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --apply
+```
+
+Apply mode does **not** modify `references_cache`. It writes the enriched entry
+to a separate research cache at `~/.cache/linkml-reference-validator/private`
+by default. Agents may inspect that cache for background context, but ordinary
+validation never reads it and never cites it as evidence. Validation therefore
+has the same inputs locally, in CI, and on another contributor's machine.
+
+To keep the research cache in a private repository or another secured location:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --private-cache-dir /path/to/private-reference-cache \
+  --apply
+```
+
+The same location can be configured for all commands:
+
+```yaml
+cache_dir: references_cache
+private_cache_dir: /path/to/private-reference-cache
+```
+
+Zotero-indexed text is preferred. If Zotero has not indexed an attachment, the
+provider downloads it from Zotero's local PDF endpoint and sends it through the
+normal PDF extractor. The source attachment key and `user_library` access type
+are persisted, but the ephemeral localhost URL is not.
+
+!!! warning "Keep research caches private"
+
+    Private cache entries contain extracted manuscript text and may contain a
+    copied PDF. Do not place this cache in a public repository or share it
+    unless you have permission to redistribute that content.
+
+Do not add `zotero` to `full_text_providers`. Even if it is configured there,
+ordinary validation rejects its `user_library` result and continues to public
+providers. Use `cache enrich` when you want to search Zotero.
+
+Support for reproducible validation backed by private manuscripts is a future,
+separate feature. The proposed design is a trusted offline job that emits a
+minimal signed excerpt attestation into the public cache. The validator would
+verify the signature using a checked-in public key; neither agents nor CI would
+have the signing key or permission to generate those files directly.
+
+## Export a cache to a Zotero collection
+
+Export references that still need full text as metadata-only CSL JSON:
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --format csl-json \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The default `--needs-full-text` mode excludes cache entries that already contain
+full text. It also excludes non-publication identifiers, deduplicates by
+normalized DOI and then PMID, and exports only an explicit metadata allowlist:
+title, authors, journal, year, DOI, PMID, and PubMed URL. Cached article text,
+excerpts, provenance, PDFs, and local paths are never exported. Use `--all` only
+when you deliberately want eligible records that already have full text.
+
+The exporter refuses to replace an existing output file unless `--force` is
+given. Review the reported counts, then in Zotero choose **File → Import → A
+file**, select the JSON file, and place the imported items in a project-specific
+collection. Zotero supports CSL JSON as a standard import format. After import,
+select the collection and run **Find Full Text**, then rerun `cache enrich`.
+
+See [Zotero's standardized-format import instructions](https://www.zotero.org/support/kb/importing_standardized_formats).
+For the complete workflow with screenshots, see
+[Export a Reference Cache to Zotero](export-cache-to-zotero.md).
+
 ## Preprints
 
 Preprints are first-class references. They are increasingly cited for early

docs/reference/cli.md

@@ -761,6 +761,69 @@ linkml-reference-validator cache lookup PMID:16888623 --content
 linkml-reference-validator cache lookup PMID:16888623 --no-cache
 ```
 
+## cache export
+
+Export public bibliographic metadata to CSL JSON for Zotero import.
+
+```bash
+linkml-reference-validator cache export --output project-zotero.json [OPTIONS]
+```
+
+**Options:**
+
+- `--output PATH, -o PATH` - Required destination file
+- `--format TEXT` - Export format (default and currently supported: `csl-json`)
+- `--needs-full-text` - Export only publication records needing full text (default)
+- `--all` - Include eligible records that already contain full text
+- `--cache-dir PATH, -c PATH` - Public reference cache directory
+- `--force, -f` - Replace an existing output file
+- `--config PATH` - Validation configuration file
+- `--verbose, -v` - Enable detailed logging
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The output is a DOI/PMID-deduplicated metadata allowlist. It never includes
+cached article content, excerpts, PDFs, local paths, or private-cache data.
+
+---
+
+## cache enrich
+
+Inventory or enrich existing abstract-only cache entries through one full-text
+provider. This is primarily intended for opt-in private-library providers such
+as Zotero.
+
+```bash
+linkml-reference-validator cache enrich [OPTIONS]
+```
+
+**Options:**
+
+- `--provider TEXT` - Registered provider name (default: `zotero`)
+- `--cache-dir PATH, -c PATH` - Reference cache directory
+- `--private-cache-dir PATH` - Separate private research-cache destination (default: `~/.cache/linkml-reference-validator/private`)
+- `--dry-run` - Report matches without changing files (default)
+- `--apply` - Materialize usable matches into the private research cache
+- `--config PATH` - Validation configuration file
+- `--verbose, -v` - Enable detailed logging
+
+```bash
+# Safe inventory
+linkml-reference-validator cache enrich --provider zotero --dry-run
+
+# Apply reviewed exact matches
+linkml-reference-validator cache enrich --provider zotero --apply
+```
+
+The public source cache is never modified by this command, and validation never
+reads the private destination. Private-library content may be copyrighted; keep
+the research cache private.
+
 ---
 
 ## Reference ID Formats