fix: skip GLRD releases not yet published on GitHub

docs/contributing/documentation/aggregation-architecture.md

@@ -103,7 +103,33 @@ flowchart TD
 - Handles root files separately from docs directory
 - Provides commit hash for reproducible builds
 
-### 2. Transform Stage (`transformer.py`)
+### 2. GitHub HTTP client (`github_api.py`)
+
+**Purpose:** Fetch GitHub release data for use by the release generators.
+
+Uses `GITHUB_TOKEN` when set. Hard-fails on any error (network, non-2xx status,
+rate-limit exhaustion, or empty first page) so release docs never ship in a
+silently degraded state.
+
+### 3. GLRD subprocess wrapper (`glrd.py`)
+
+**Purpose:** Query active minor-release versions and release metadata from the
+`glrd` command-line tool. The data drives the release-table generator.
+
+Returns `None` on any failure rather than raising exceptions.
+
+### 4. Sphinx Markdown builder (`sphinx_builder.py`)
+
+**Purpose:** Produce plain Markdown output from Sphinx-based documentation so
+it can be consumed by VitePress through the normal aggregation pipeline.
+
+Invokes `python -m sphinx -M markdown` on a fetched repository and copies the
+resulting Markdown into the aggregation output directory. Called only for repos
+configured with `"structure": "sphinx"` in `repos-config.json`. Requires
+`sphinx`, `sphinx-markdown-builder`, and any project-specific Sphinx extensions
+installed in the same Python environment as the aggregator.
+
+### 5. Transform Stage (`transformer.py`)
 
 **Purpose:** Modify content to work in the aggregated site
 
@@ -121,7 +147,7 @@ flowchart TD
    - Quote YAML values safely
    - Preserve existing metadata
 
-### 3. Structure Stage (`structure.py`)
+### 6. Structure Stage (`structure.py`)
 
 **Purpose:** Organize documentation into the final directory structure
 
@@ -135,6 +161,28 @@ flowchart TD
 3. **Media Copying:** Discover and copy media directories
 4. **Markdown Processing:** Apply front-matter fixes to all copied files
 
+### 7. Release-table generator (`releases.py`)
+
+**Purpose:** Build the release-status table from GLRD data, filtered against
+fetched GitHub tags.
+
+GLRD-listed rows that carry a `minor` version component are only emitted when
+their normalized version string (`{major}.{minor}[.{patch}]`, leading `v`
+stripped) appears in the fetched GitHub tag set. Major-only rows are always
+emitted. Each skipped row logs a one-line warning to `stderr`.
+
+### 8. Per-release notes generator (`release_notes.py`)
+
+**Purpose:** Write one Markdown page per pre-fetched GitHub release. Makes no
+network calls; accepts the release list returned by
+`github_api.list_repo_releases()`.
+
+### 9. Flavor matrix generator (`flavor_matrix.py`)
+
+**Purpose:** Generate flavor matrix documentation from `flavors.yaml` and
+feature metadata. Parses `flavors.yaml` directly using the Garden Linux
+`FlavorsParser` and `FeaturesParser` classes.
+
 ## Key Mechanisms
 
 ### Targeted Documentation
@@ -262,7 +310,7 @@ Temp Directory                 Docs Output
 ### Structure Stage
 
 - **Targeted copy:** O(n) where n = files with github_target_path
-- **Link verification:** O(n * l) where l = avg links per file
+- **Link verification:** O(n \* l) where l = avg links per file
 - **Media copy:** O(m) where m = media files
 
 ### Overall
@@ -291,6 +339,13 @@ Temp Directory                 Docs Output
 - Conflicting file paths → Error with clear message
 - Media directory not found → Log warning, continue
 
+### GitHub API Failures
+
+Any GitHub API failure during release pre-fetch (network, non-2xx status,
+rate-limit exhaustion, or empty result) is fatal. This prevents shipping
+silently degraded release documentation. See
+[GitHub API token](./working-locally.md#github-api-token) for token setup.
+
 ## Related Topics
 
 <RelatedTopics />

docs/contributing/documentation/technical.md

@@ -27,11 +27,18 @@ src/
 ├── migration_tracker.py  # Standalone utility
 └── aggregation/          # Core package
     ├── __init__.py
-    ├── models.py         # Data classes
     ├── config.py         # Config I/O
+    ├── constants.py      # Shared constants (Mermaid theme, GitHub URL templates)
     ├── fetcher.py        # Git + local fetch
-    ├── transformer.py    # Content transforms
-    └── structure.py      # Directory transforms
+    ├── flavor_matrix.py  # Flavor matrix docs from flavors.yaml
+    ├── github_api.py     # GitHub HTTP client (releases fetch, auth)
+    ├── glrd.py           # GLRD subprocess wrapper (active versions, metadata)
+    ├── models.py         # Data classes
+    ├── release_notes.py  # Per-release note pages from GH data
+    ├── releases.py       # Release tables from GLRD (filtered by GH tags)
+    ├── sphinx_builder.py # Sphinx-to-Markdown builder for sphinx-structured repos
+    ├── structure.py      # Directory transforms
+    └── transformer.py    # Content transforms
 ```
 
 ## Module Reference
@@ -50,6 +57,18 @@ Configuration file handling:
 - **`load_config()`** — Parse repos-config.json
 - **`save_config()`** — Write updated config (commit locks)
 
+### `aggregation/constants.py`
+
+Shared constants for release-doc generation. Contains no functions; exposes
+only module-level constants:
+
+- **`GANTT_THEME`** — Mermaid Gantt chart theme variables using the Garden
+  Linux color palette.
+- **`GITHUB_BASE_URL`**, **`RELEASES_TAG_URL`**, **`COMMITS_URL`** — GitHub URL
+  templates for the `gardenlinux/gardenlinux` repository.
+- **`LIFECYCLE_LINKS`** — Anchor-link map for release lifecycle documentation
+  sections.
+
 ### `aggregation/fetcher.py`
 
 Repository fetching:
@@ -77,19 +96,99 @@ Content transformation:
 - **`parse_frontmatter()`** — Extract metadata from markdown front-matter
 - **`fix_broken_project_links()`** — Validate and fix links to project mirrors
 
+### `aggregation/github_api.py`
+
+GitHub HTTP client for release data:
+
+- **`GitHubAPIError`** — Exception raised on any GitHub API failure (network,
+  non-2xx, rate-limit, or empty first page).
+- **`get_json(url)`** — Fetch a single GitHub API URL and return the parsed
+  JSON response. Uses `GITHUB_TOKEN` when set. Hard-fails on non-2xx status.
+- **`list_repo_releases(owner, repo, per_page=100)`** — Paginate through all
+  releases for a repository and return the full list. Paginates until GitHub
+  returns an empty page (uncapped). Hard-fails on any error or an empty first
+  page.
+
+This module is the single source of truth for GitHub HTTP calls in the
+aggregation package. Future callers should use it rather than adding new
+ad-hoc HTTP requests.
+
+### `aggregation/glrd.py`
+
+GLRD subprocess wrapper:
+
+- **`run_glrd_json(args)`** — Run `glrd` with JSON output and return the parsed
+  data. Returns `None` on any failure (binary not found, non-zero exit, JSON
+  decode error).
+- **`get_active_minor_versions()`** — Return the set of active minor-release
+  version strings from GLRD (e.g. `{"1877.14", "2150.1.0"}`).
+
+### `aggregation/sphinx_builder.py`
+
+Sphinx-to-Markdown builder for repos configured with
+`"structure": "sphinx"` in `repos-config.json`:
+
+- **`build_sphinx_markdown(repo_dir, docs_path, output_dir, target_map=None)`**
+  — Run `python -m sphinx -M markdown` on a fetched repository and copy the
+  resulting Markdown into `output_dir` so the standard Transform/Structure
+  stages can consume it. Injects VitePress frontmatter, carries over
+  hand-written Markdown files that have a `github_target_path` frontmatter
+  field, and strips Sphinx HTML anchors that break VitePress compatibility.
+  Returns `True` on success, `False` on any failure. Requires `sphinx`,
+  `sphinx-markdown-builder`, and any project-specific Sphinx extensions
+  installed in the same Python environment as the aggregator.
+
+### `aggregation/releases.py`
+
+Release-table generation from GLRD data:
+
+- **`generate_release_docs(docs_dir, existing_gh_tags)`** — Read GLRD release
+  data and write per-release documentation pages. Only writes pages for
+  releases whose normalized version string appears in `existing_gh_tags`.
+- **`generate_release_table(releases_data, active_versions,
+existing_gh_tags)`** — Build the release-status table. GLRD-listed rows that
+  carry a `minor` version component are only emitted when their normalized
+  version string appears in `existing_gh_tags`. Major-only rows are always
+  emitted. Skipped rows are logged to `stderr`.
+
+### `aggregation/release_notes.py`
+
+Per-release note page generation:
+
+- **`generate_release_notes_docs(docs_dir, releases)`** — Write one Markdown
+  page per release from the pre-fetched `releases` list. Accepts the release
+  list returned by `github_api.list_repo_releases()` and makes no network
+  calls itself.
+
+### `aggregation/flavor_matrix.py`
+
+Flavor matrix documentation generator:
+
+- **`get_flavor_list(gardenlinux_repo_dir)`** — Parse `flavors.yaml` in the
+  fetched Garden Linux repository and return a per-architecture dict of flavor
+  combinations. Returns `None` on failure.
+- **`generate_flavor_matrix_docs(docs_dir, gardenlinux_repo_dir)`** — Generate
+  the flavor matrix Markdown page by combining `get_flavor_list()` output with
+  feature metadata from the Garden Linux `FeaturesParser`. Appends the
+  generated table to the existing aggregated `reference/flavor-matrix.md` file.
+  Returns `True` on success, `False` on any failure.
+
 ### `aggregation/structure.py`
 
 Directory operations:
 
-- **`transform_directory_structure()`** — Restructure docs based on config
-  mapping
 - **`copy_targeted_docs(source_dir, docs_dir, repo_name, media_dirs=None, root_files=None)`**
   — Copy files with `github_target_path` front-matter to specified locations
   - Handles nested media dirs (e.g., `tutorials/assets/`) by copying to same
     relative path
   - Handles root-level media dirs (e.g., `_static/`) by copying to common
     ancestor of targeted files
   - Supports scanning root_files for targeted placement
+- **`verify_internal_links(source_dir, docs_dir, repo_name)`** — Verify that
+  all internal relative links in shipped Markdown files resolve to files that
+  were also shipped. Returns the number of broken links found (0 = success).
+  Hard-fails aggregation when any shipped file links to a source-repo file that
+  was not itself shipped.
 - **`process_markdown_file()`** — Transform single markdown file (links,
   front-matter)
 - **`process_all_markdown()`** — Batch process all markdown files in directory
@@ -180,7 +279,7 @@ To add a new transformation:
 
 To add a new structure mapping type:
 
-1. Update `transform_directory_structure()` in `structure.py`
+1. Add the corresponding logic to `structure.py`
 2. Add corresponding structure key handling
 3. Update configuration documentation
 

docs/contributing/documentation/working-locally.md

@@ -62,6 +62,18 @@ make aggregate
 This fetches documentation from the configured repositories at their locked
 commit hashes.
 
+#### GitHub API token
+
+`make aggregate` also fetches every release from `gardenlinux/gardenlinux` via
+the GitHub Releases API to generate per-release note pages and to validate
+GLRD-listed minor releases. Set `GITHUB_TOKEN` before running `make aggregate`
+to prevent rate-limiting:
+
+```bash
+export GITHUB_TOKEN=$(gh auth token)
+make aggregate
+```
+
 #### From Local Repositories (Development)
 
 For local development, use `repos-config.local.json` with `file://` URLs:
@@ -175,6 +187,20 @@ python3 --version  # Should be 3.x
 Check that `repos-config.json` or `repos-config.local.json` is properly
 configured. See the [configuration reference](./configuration.md) for details.
 
+### GitHub API rate limit or fetch error
+
+If `make aggregate` fails with a message like
+`Could not fetch GitHub releases — HTTP 403 …`, the GitHub API rate limit is
+exhausted or the request was rejected. Set `GITHUB_TOKEN` and retry:
+
+```bash
+export GITHUB_TOKEN=$(gh auth token)
+make aggregate
+```
+
+See the [GitHub API token](#github-api-token) note under Step 3 for rate-limit
+details.
+
 ## Related Topics
 
 <RelatedTopics />

src/aggregate.py

@@ -15,6 +15,7 @@
                           save_config)
 from aggregation.structure import verify_internal_links
 from aggregation.flavor_matrix import generate_flavor_matrix_docs
+from aggregation.github_api import GitHubAPIError, list_repo_releases
 from aggregation.release_notes import generate_release_notes_docs
 from aggregation.releases import generate_release_docs
 
@@ -290,17 +291,36 @@ def main() -> int:
             print(f"{'='*60}\n")
             generate_flavor_matrix_docs(docs_dir, gardenlinux_temp_dir)
 
+    # Fetch all GitHub releases up front — used by both release-doc generators.
+    # Hard-fail here rather than inside the generators so the earlier repo-
+    # aggregation results are still printed before we exit non-zero.
+    print(f"\n{'='*60}")
+    print("Fetching GitHub releases...")
+    print(f"{'='*60}\n")
+    try:
+        gh_releases = list_repo_releases("gardenlinux", "gardenlinux")
+    except GitHubAPIError as exc:
+        print(
+            f"\nError: Could not fetch GitHub releases — {exc}\n"
+            "Hint: set GITHUB_TOKEN to avoid rate-limit issues: "
+            "export GITHUB_TOKEN=$(gh auth token)",
+            file=sys.stderr,
+        )
+        return 1
+    existing_gh_tags = {r["tag_name"].lstrip("v") for r in gh_releases}
+    print(f"  Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)")
+
     # Generate release documentation from GLRD
     print(f"\n{'='*60}")
     print("Generating release documentation...")
     print(f"{'='*60}\n")
-    generate_release_docs(docs_dir)
+    generate_release_docs(docs_dir, existing_gh_tags)
 
-    # Generate release notes from GitHub
+    # Generate release notes from pre-fetched GitHub releases
     print(f"\n{'='*60}")
-    print("Fetching release notes from GitHub...")
+    print("Generating release notes from GitHub...")
     print(f"{'='*60}\n")
-    generate_release_notes_docs(docs_dir)
+    generate_release_notes_docs(docs_dir, gh_releases)
 
     # Summary
     print(f"\n{'='*60}")