Add remote-docs collection to the build pipeline via a definition file

[MariusStorhaug]

Context and Request

The PSModule docs site at psmodule.io/docs is built from this repository. Documentation for sub-projects — such as Process-PSModule — is authored and maintained inside those projects' own repositories. At the moment there is no way to include that content in the published site without manually copying and committing it here, which creates drift, duplication, and maintenance overhead.

A maintainer wants to declare, in a single definition file, how remote docs folders map to locations in this site — for example: "the docs/ folder in PSModule/Process-PSModule becomes docs/The-Process/ in the published site". During the CI build, a pre-build step reads the definition file, fetches the declared content from each source repository, and places it in the target location before zensical build runs. The fetched content is ephemeral and is never committed to this repository.

Acceptance criteria

• A definition file in this repository declares one or more remote source mappings (source repo, source path, destination path, optional ref).
• The CI build fetches the declared sources before running zensical build — no manual step is required.
• Fetched content appears in the published site under the declared destination paths.
• No fetched content is ever committed to this repository (paths are .gitignored).
• A missing or unreachable source fails the build with a descriptive error.
• Adding a new remote source requires only an edit to the definition file and the zensical.toml nav — no workflow changes.

---

Technical Decisions

Definition file format and location
A dedicated remote-sources.yml at the repository root. YAML is human-readable, supports inline comments, and is already used widely across PSModule repositories. Embedding the configuration in zensical.toml was considered but rejected: Zensical would need custom extension support, and a standalone file keeps the fetch concern cleanly separated from the site build configuration.

Proposed schema:

sources:
  - repo: PSModule/Process-PSModule
    ref: main                     # optional; defaults to the repo's default branch
    path: docs                    # source path within the remote repo (file or directory)
    destination: docs/The-Process # target path relative to this repo root

Fetch mechanism
A PowerShell script at .github/scripts/Get-RemoteDocs.ps1 reads remote-sources.yml and downloads each declared source. The recommended approach is to download the repo archive via the GitHub API (GET /repos/{owner}/{repo}/tarball/{ref}) and extract only the declared sub-path to the destination. This keeps the implementation self-contained, avoids additional git operations, and works for both file and directory sources. Using Invoke-GitHubAPI (already present in Update-Index.ps1) ensures consistent authentication handling.

Integration point
A new step is inserted in .github/workflows/Docs.yml between "Install Zensical" and "Build Zensical project". The step runs Get-RemoteDocs.ps1 using PSModule/GitHub-Script. Because the step runs inside the Actions runner, the GITHUB_TOKEN already available in the environment is sufficient for public PSModule repositories.

Gitignore
Destination paths declared in remote-sources.yml must be added to .gitignore to prevent accidental commits. The script can validate this on startup and warn if a destination is not excluded.

---

Implementation Plan

• Create remote-sources.yml at the repository root with the PSModule/Process-PSModule → docs/The-Process entry as the first example.
• Add the declared destination paths to .gitignore.
• Write .github/scripts/Get-RemoteDocs.ps1:
  • Reads remote-sources.yml.
  • For each source: downloads the tarball, extracts the declared sub-path, copies files to the destination, logs progress.
  • Fails with a descriptive error if a source is unreachable or the destination is not .gitignored.
• Add the pre-build step to .github/workflows/Docs.yml.
• Update zensical.toml nav to reference the collected content paths (e.g., The-Process/index.md).
• Remove the docs/Process/ placeholder page and the Update-Index.ps1 Process README section introduced as a temporary workaround (superseded by this approach).
• Validate the end-to-end build: fetched content is present in the published site and absent from committed files.