diff --git a/docs/contributing/documentation/aggregation-architecture.md b/docs/contributing/documentation/aggregation-architecture.md
index b9272a5..9df76c9 100644
--- a/docs/contributing/documentation/aggregation-architecture.md
+++ b/docs/contributing/documentation/aggregation-architecture.md
@@ -103,7 +103,33 @@ flowchart TD
- Handles root files separately from docs directory
- Provides commit hash for reproducible builds
-### 2. Transform Stage (`transformer.py`)
+### 2. GitHub HTTP client (`github_api.py`)
+
+**Purpose:** Fetch GitHub release data for use by the release generators.
+
+Uses `GITHUB_TOKEN` when set. Hard-fails on any error (network, non-2xx status,
+rate-limit exhaustion, or empty first page) so release docs never ship in a
+silently degraded state.
+
+### 3. GLRD subprocess wrapper (`glrd.py`)
+
+**Purpose:** Query active minor-release versions and release metadata from the
+`glrd` command-line tool. The data drives the release-table generator.
+
+Returns `None` on any failure rather than raising exceptions.
+
+### 4. Sphinx Markdown builder (`sphinx_builder.py`)
+
+**Purpose:** Produce plain Markdown output from Sphinx-based documentation so
+it can be consumed by VitePress through the normal aggregation pipeline.
+
+Invokes `python -m sphinx -M markdown` on a fetched repository and copies the
+resulting Markdown into the aggregation output directory. Called only for repos
+configured with `"structure": "sphinx"` in `repos-config.json`. Requires
+`sphinx`, `sphinx-markdown-builder`, and any project-specific Sphinx extensions
+installed in the same Python environment as the aggregator.
+
+### 5. Transform Stage (`transformer.py`)
**Purpose:** Modify content to work in the aggregated site
@@ -121,7 +147,7 @@ flowchart TD
- Quote YAML values safely
- Preserve existing metadata
-### 3. Structure Stage (`structure.py`)
+### 6. Structure Stage (`structure.py`)
**Purpose:** Organize documentation into the final directory structure
@@ -135,6 +161,28 @@ flowchart TD
3. **Media Copying:** Discover and copy media directories
4. **Markdown Processing:** Apply front-matter fixes to all copied files
+### 7. Release-table generator (`releases.py`)
+
+**Purpose:** Build the release-status table from GLRD data, filtered against
+fetched GitHub tags.
+
+GLRD-listed rows that carry a `minor` version component are only emitted when
+their normalized version string (`{major}.{minor}[.{patch}]`, leading `v`
+stripped) appears in the fetched GitHub tag set. Major-only rows are always
+emitted. Each skipped row logs a one-line warning to `stderr`.
+
+### 8. Per-release notes generator (`release_notes.py`)
+
+**Purpose:** Write one Markdown page per pre-fetched GitHub release. Makes no
+network calls; accepts the release list returned by
+`github_api.list_repo_releases()`.
+
+### 9. Flavor matrix generator (`flavor_matrix.py`)
+
+**Purpose:** Generate flavor matrix documentation from `flavors.yaml` and
+feature metadata. Parses `flavors.yaml` directly using the Garden Linux
+`FlavorsParser` and `FeaturesParser` classes.
+
## Key Mechanisms
### Targeted Documentation
@@ -262,7 +310,7 @@ Temp Directory Docs Output
### Structure Stage
- **Targeted copy:** O(n) where n = files with github_target_path
-- **Link verification:** O(n * l) where l = avg links per file
+- **Link verification:** O(n \* l) where l = avg links per file
- **Media copy:** O(m) where m = media files
### Overall
@@ -291,6 +339,13 @@ Temp Directory Docs Output
- Conflicting file paths → Error with clear message
- Media directory not found → Log warning, continue
+### GitHub API Failures
+
+Any GitHub API failure during release pre-fetch (network, non-2xx status,
+rate-limit exhaustion, or empty result) is fatal. This prevents shipping
+silently degraded release documentation. See
+[GitHub API token](./working-locally.md#github-api-token) for token setup.
+
## Related Topics
diff --git a/docs/contributing/documentation/ci-workflows-reference.md b/docs/contributing/documentation/ci-workflows-reference.md
index 696022d..599ec37 100644
--- a/docs/contributing/documentation/ci-workflows-reference.md
+++ b/docs/contributing/documentation/ci-workflows-reference.md
@@ -19,11 +19,11 @@ related_topics:
## 1. `docs-check.yml` — Per-Repo Caller Workflow
-| | |
-|---|---|
-| **File** | `.github/workflows/docs-check.yml` |
-| **Lives in** | Each aggregated repository |
-| **Type** | Caller workflow — invokes the reusable `docs-checks.yml` from `docs-ng` |
+| | |
+| ------------ | ----------------------------------------------------------------------- |
+| **File** | `.github/workflows/docs-check.yml` |
+| **Lives in** | Each aggregated repository |
+| **Type** | Caller workflow — invokes the reusable `docs-checks.yml` from `docs-ng` |
### Triggers
@@ -33,12 +33,12 @@ on:
branches: [main, docs-ng]
types: [opened, synchronize, reopened, closed]
paths:
- - 'docs/**'
- - 'features/*/README.md'
- - 'features/*/info.yaml'
- - 'flavors.yaml'
- - 'CONTRIBUTING.md'
- - 'SECURITY.md'
+ - "docs/**"
+ - "features/*/README.md"
+ - "features/*/info.yaml"
+ - "flavors.yaml"
+ - "CONTRIBUTING.md"
+ - "SECURITY.md"
push:
branches: [main, docs-ng]
paths:
@@ -51,18 +51,18 @@ can detect merged PRs.
### Jobs
-| Job | Condition | Purpose |
-|-----|-----------|---------|
-| `docs-checks` | Skipped when `action == 'closed'` | Calls the reusable quality-check workflow in `docs-ng` with override inputs |
+| Job | Condition | Purpose |
+| ---------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
+| `docs-checks` | Skipped when `action == 'closed'` | Calls the reusable quality-check workflow in `docs-ng` with override inputs |
| `notify-docs-ng` | Runs after `docs-checks` succeeds **or** when a PR is merged (always-conditional) | Generates a GitHub App token and sends a `repository_dispatch` to `docs-ng` |
### Inputs Passed to Reusable Workflow
-| Input | Value |
-|-------|-------|
-| `override-repo` | Hard-coded repository name (e.g., `gardenlinux`) |
-| `override-ref` | `github.head_ref` (PR branch) or `github.ref_name` (push) |
-| `override-commit` | `pull_request.head.sha` or `github.sha` |
+| Input | Value |
+| ----------------- | --------------------------------------------------------- |
+| `override-repo` | Hard-coded repository name (e.g., `gardenlinux`) |
+| `override-ref` | `github.head_ref` (PR branch) or `github.ref_name` (push) |
+| `override-commit` | `pull_request.head.sha` or `github.sha` |
### Dispatch Payload Sent to `docs-ng`
@@ -78,9 +78,9 @@ can detect merged PRs.
### Required Secrets
-| Secret | Purpose |
-|--------|---------|
-| `DOCS_BOT_APP_ID` | GitHub App identifier for `gardenlinux-docs-bot` |
+| Secret | Purpose |
+| ---------------------- | --------------------------------------------------------- |
+| `DOCS_BOT_APP_ID` | GitHub App identifier for `gardenlinux-docs-bot` |
| `DOCS_BOT_PRIVATE_KEY` | Private key used to sign the JWT for App token generation |
### Outputs / Side-Effects
@@ -110,20 +110,19 @@ flowchart TD
### Common Failure Causes
-| Symptom | Likely cause |
-|---------|--------------|
-| `docs-checks` job fails | Broken links, spelling errors, or woke violations in the documentation change |
+| Symptom | Likely cause |
+| -------------------------- | -------------------------------------------------------------------------------------------------- |
+| `docs-checks` job fails | Broken links, spelling errors, or woke violations in the documentation change |
| `notify-docs-ng` job fails | `DOCS_BOT_APP_ID` or `DOCS_BOT_PRIVATE_KEY` secret missing/invalid; App not installed on `docs-ng` |
-| Workflow not triggered | Changed files do not match the `paths:` filter |
-
+| Workflow not triggered | Changed files do not match the `paths:` filter |
## 2. `docs-checks.yml` — Reusable Quality Checks Workflow
-| | |
-|---|---|
-| **File** | `.github/workflows/docs-checks.yml` |
-| **Lives in** | `gardenlinux/docs-ng` |
-| **Type** | Reusable workflow (`workflow_call`) + own PR/push trigger |
+| | |
+| ------------ | --------------------------------------------------------- |
+| **File** | `.github/workflows/docs-checks.yml` |
+| **Lives in** | `gardenlinux/docs-ng` |
+| **Type** | Reusable workflow (`workflow_call`) + own PR/push trigger |
### Triggers
@@ -132,15 +131,15 @@ on:
workflow_call:
inputs:
override-repo:
- description: 'Repository name to override (e.g., gardenlinux)'
+ description: "Repository name to override (e.g., gardenlinux)"
required: false
type: string
override-ref:
- description: 'Git ref to use for the overridden repo'
+ description: "Git ref to use for the overridden repo"
required: false
type: string
override-commit:
- description: 'Git commit SHA to use for the overridden repo'
+ description: "Git commit SHA to use for the overridden repo"
required: false
type: string
pull_request:
@@ -155,11 +154,11 @@ own documentation (with no overrides — all repos use their pinned commits).
### Inputs
-| Input | Required | Description |
-|-------|----------|-------------|
-| `override-repo` | no | Repository name in `repos-config.json` to override |
-| `override-ref` | no | Git ref (branch) to check out for the overridden repo |
-| `override-commit` | no | Exact commit SHA to aggregate from |
+| Input | Required | Description |
+| ----------------- | -------- | ----------------------------------------------------- |
+| `override-repo` | no | Repository name in `repos-config.json` to override |
+| `override-ref` | no | Git ref (branch) to check out for the overridden repo |
+| `override-commit` | no | Exact commit SHA to aggregate from |
When all three inputs are provided, `aggregate.py` substitutes the pinned
commit in `repos-config.json` with the specified commit, effectively testing
@@ -167,18 +166,18 @@ the PR's documentation against the rest of the published site.
### Jobs
-| Job | Depends on | Runner | Purpose |
-|-----|-----------|--------|---------|
-| `aggregate` | — | `ubuntu-24.04` | Runs `aggregate.py` with override arguments; uploads `docs/` as artifact |
-| `linkcheck` | `aggregate` | `ubuntu-24.04` | Runs lychee link checker on the aggregated `docs/**/*.md` |
-| `spelling` | `aggregate` | `ubuntu-24.04` | Runs `make spelling` (codespell) |
-| `woke` | `aggregate` | `ubuntu-24.04` | Runs `make woke` (inclusive-language checker) |
+| Job | Depends on | Runner | Purpose |
+| ----------- | ----------- | -------------- | ------------------------------------------------------------------------ |
+| `aggregate` | — | `ubuntu-24.04` | Runs `aggregate.py` with override arguments; uploads `docs/` as artifact |
+| `linkcheck` | `aggregate` | `ubuntu-24.04` | Runs lychee link checker on the aggregated `docs/**/*.md` |
+| `spelling` | `aggregate` | `ubuntu-24.04` | Runs `make spelling` (codespell) |
+| `woke` | `aggregate` | `ubuntu-24.04` | Runs `make woke` (inclusive-language checker) |
### Artifacts
-| Artifact | Contents | Retention |
-|----------|----------|-----------|
-| `aggregated-docs` | Full `docs/` directory after aggregation | 1 day |
+| Artifact | Contents | Retention |
+| ----------------- | ---------------------------------------- | --------- |
+| `aggregated-docs` | Full `docs/` directory after aggregation | 1 day |
The artifact is consumed by the three parallel lint jobs (`linkcheck`,
`spelling`, `woke`) to avoid re-running aggregation.
@@ -189,6 +188,10 @@ No secrets are required by this workflow itself (it only reads public
repositories). When called as a reusable workflow, it inherits the caller's
`GITHUB_TOKEN` permissions (read-only is sufficient).
+:::tip `GITHUB_TOKEN`
+Setting the `GITHUB_TOKEN` environment variable is recommended to prevent rate limiting.
+:::
+
### Job Graph
```mermaid
@@ -207,20 +210,20 @@ flowchart TD
### Common Failure Causes
-| Symptom | Likely cause |
-|---------|--------------|
+| Symptom | Likely cause |
+| --------------------- | ------------------------------------------------------------------------------------------------------------ |
| `aggregate` job fails | Invalid `repos-config.json` entry for the overridden repo; commit SHA not reachable; Python dependency issue |
-| `linkcheck` job fails | Broken internal or external links in Markdown files; check lychee output for specific URLs |
-| `spelling` job fails | Unknown words — add them to the project's codespell dictionary (`.codespell/project-words.txt`) |
-| `woke` job fails | Non-inclusive language detected — see the woke rule file (`.woke.yml`) for alternatives |
+| `linkcheck` job fails | Broken internal or external links in Markdown files; check lychee output for specific URLs |
+| `spelling` job fails | Unknown words — add them to the project's codespell dictionary (`.codespell/project-words.txt`) |
+| `woke` job fails | Non-inclusive language detected — see the woke rule file (`.woke.yml`) for alternatives |
## 3. `docs-pr.yml` — Automated Docs PR Workflow
-| | |
-|---|---|
-| **File** | `.github/workflows/docs-pr.yml` |
-| **Lives in** | `gardenlinux/docs-ng` |
-| **Type** | `repository_dispatch` consumer + `workflow_dispatch` (manual) |
+| | |
+| ------------ | ------------------------------------------------------------- |
+| **File** | `.github/workflows/docs-pr.yml` |
+| **Lives in** | `gardenlinux/docs-ng` |
+| **Type** | `repository_dispatch` consumer + `workflow_dispatch` (manual) |
### Triggers
@@ -230,11 +233,11 @@ on:
types: [docs-pr]
workflow_dispatch:
inputs:
- repo: # required, string
- pr_number: # required, string
- commit_sha: # required, string
- ref: # required, string
- event: # required, choice: [pr_success, merged]
+ repo: # required, string
+ pr_number: # required, string
+ commit_sha: # required, string
+ ref: # required, string
+ event: # required, choice: [pr_success, merged]
```
The workflow is normally triggered by `repository_dispatch` from a source
@@ -245,18 +248,18 @@ repo's `docs-check.yml`. It can also be triggered manually via
The `client_payload` (or `inputs` for manual dispatch) must contain:
-| Field | Type | Description |
-|-------|------|-------------|
-| `repo` | string | Repository name as it appears in `repos-config.json` (e.g., `gardenlinux`) |
-| `pr_number` | string | PR number in the source repository |
-| `commit_sha` | string | Commit SHA to pin in `repos-config.json` |
-| `ref` | string | Branch name — feature branch for open PRs, base branch (e.g., `main`) for merged PRs |
-| `event` | string | Either `pr_success` (PR is open, checks passed) or `merged` (PR was merged) |
+| Field | Type | Description |
+| ------------ | ------ | ------------------------------------------------------------------------------------ |
+| `repo` | string | Repository name as it appears in `repos-config.json` (e.g., `gardenlinux`) |
+| `pr_number` | string | PR number in the source repository |
+| `commit_sha` | string | Commit SHA to pin in `repos-config.json` |
+| `ref` | string | Branch name — feature branch for open PRs, base branch (e.g., `main`) for merged PRs |
+| `event` | string | Either `pr_success` (PR is open, checks passed) or `merged` (PR was merged) |
### Jobs
-| Job | Runner | Purpose |
-|-----|--------|---------|
+| Job | Runner | Purpose |
+| ----------- | -------------- | --------------------------------------------------------------------------------------------------------- |
| `create-pr` | `ubuntu-24.04` | Creates/updates a branch, modifies `repos-config.json`, opens/updates a PR, and comments on the source PR |
### Job Steps (in order)
@@ -280,19 +283,19 @@ The `client_payload` (or `inputs` for manual dispatch) must contain:
### Required Secrets
-| Secret | Purpose |
-|--------|---------|
-| `DOCS_BOT_APP_ID` | GitHub App identifier (configured at org level in `docs-ng`) |
-| `DOCS_BOT_PRIVATE_KEY` | Private key for JWT token generation |
+| Secret | Purpose |
+| ---------------------- | ------------------------------------------------------------ |
+| `DOCS_BOT_APP_ID` | GitHub App identifier (configured at org level in `docs-ng`) |
+| `DOCS_BOT_PRIVATE_KEY` | Private key for JWT token generation |
### Outputs / Side-Effects
-| Side-effect | Details |
-|-------------|---------|
-| Branch `auto//` | Created or force-updated in `docs-ng` |
-| Draft PR in `docs-ng` | Title: `docs(): Update commit lock from PR #` |
-| PR transitioned to ready-for-review | When `event == merged` |
-| Comment on source PR | Contains docs-ng PR URL and Netlify preview link |
+| Side-effect | Details |
+| ----------------------------------- | ------------------------------------------------------ |
+| Branch `auto//` | Created or force-updated in `docs-ng` |
+| Draft PR in `docs-ng` | Title: `docs(): Update commit lock from PR #` |
+| PR transitioned to ready-for-review | When `event == merged` |
+| Comment on source PR | Contains docs-ng PR URL and Netlify preview link |
### Branch Naming Convention
@@ -327,20 +330,20 @@ flowchart TD
### Common Failure Causes
-| Symptom | Likely cause |
-|---------|--------------|
-| Job fails at "Generate GitHub App token" | `DOCS_BOT_APP_ID` or `DOCS_BOT_PRIVATE_KEY` not set in `docs-ng` repository secrets |
-| "Repository not found in repos-config.json" | The dispatching repo's name does not match any `name` field in `repos-config.json` |
-| Force-push fails | Branch protection rules on `docs-ng` blocking the bot (auto branches should not be protected) |
-| Comment not posted | App token lacks write access to the source repo's issues/PRs |
+| Symptom | Likely cause |
+| ------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Job fails at "Generate GitHub App token" | `DOCS_BOT_APP_ID` or `DOCS_BOT_PRIVATE_KEY` not set in `docs-ng` repository secrets |
+| "Repository not found in repos-config.json" | The dispatching repo's name does not match any `name` field in `repos-config.json` |
+| Force-push fails | Branch protection rules on `docs-ng` blocking the bot (auto branches should not be protected) |
+| Comment not posted | App token lacks write access to the source repo's issues/PRs |
## 4. `check-pr-main.yml` — `repos-config.json` Validator
-| | |
-|---|---|
-| **File** | `.github/workflows/check-pr-main.yml` |
-| **Lives in** | `gardenlinux/docs-ng` |
-| **Type** | PR validator (status check) |
+| | |
+| ------------ | ------------------------------------- |
+| **File** | `.github/workflows/check-pr-main.yml` |
+| **Lives in** | `gardenlinux/docs-ng` |
+| **Type** | PR validator (status check) |
### Triggers
@@ -373,8 +376,8 @@ check fails, blocking the merge.
### Jobs
-| Job | Runner | Purpose |
-|-----|--------|---------|
+| Job | Runner | Purpose |
+| ---------- | -------------- | -------------------------------------------------------------------------- |
| `validate` | `ubuntu-24.04` | Reads `repos-config.json`, iterates over repos, validates each `ref` field |
### Required Secrets / Permissions
@@ -408,11 +411,11 @@ flowchart TD
### Common Failure Causes
-| Symptom | Likely cause |
-|---------|--------------|
-| Check fails with "non-main branch references" | The automated PR was created from an open (not yet merged) source PR — the ref still points at the feature branch |
-| Check fails after source PR was merged | Race condition — the `docs-pr.yml` workflow has not yet run to update the ref. Wait for the "merged" dispatch to complete and force-push the updated branch |
-| False positive on `docs-ng` ref | The `docs-ng` ref is in the allowed set — if validation fails, check the exact spelling in `repos-config.json` |
+| Symptom | Likely cause |
+| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Check fails with "non-main branch references" | The automated PR was created from an open (not yet merged) source PR — the ref still points at the feature branch |
+| Check fails after source PR was merged | Race condition — the `docs-pr.yml` workflow has not yet run to update the ref. Wait for the "merged" dispatch to complete and force-push the updated branch |
+| False positive on `docs-ng` ref | The `docs-ng` ref is in the allowed set — if validation fails, check the exact spelling in `repos-config.json` |
## Related Topics
diff --git a/docs/contributing/documentation/technical.md b/docs/contributing/documentation/technical.md
index 3fd8bc7..e488987 100644
--- a/docs/contributing/documentation/technical.md
+++ b/docs/contributing/documentation/technical.md
@@ -27,11 +27,18 @@ src/
├── migration_tracker.py # Standalone utility
└── aggregation/ # Core package
├── __init__.py
- ├── models.py # Data classes
├── config.py # Config I/O
+ ├── constants.py # Shared constants (Mermaid theme, GitHub URL templates)
├── fetcher.py # Git + local fetch
- ├── transformer.py # Content transforms
- └── structure.py # Directory transforms
+ ├── flavor_matrix.py # Flavor matrix docs from flavors.yaml
+ ├── github_api.py # GitHub HTTP client (releases fetch, auth)
+ ├── glrd.py # GLRD subprocess wrapper (active versions, metadata)
+ ├── models.py # Data classes
+ ├── release_notes.py # Per-release note pages from GH data
+ ├── releases.py # Release tables from GLRD (filtered by GH tags)
+ ├── sphinx_builder.py # Sphinx-to-Markdown builder for sphinx-structured repos
+ ├── structure.py # Directory transforms
+ └── transformer.py # Content transforms
```
## Module Reference
@@ -50,6 +57,18 @@ Configuration file handling:
- **`load_config()`** — Parse repos-config.json
- **`save_config()`** — Write updated config (commit locks)
+### `aggregation/constants.py`
+
+Shared constants for release-doc generation. Contains no functions; exposes
+only module-level constants:
+
+- **`GANTT_THEME`** — Mermaid Gantt chart theme variables using the Garden
+ Linux color palette.
+- **`GITHUB_BASE_URL`**, **`RELEASES_TAG_URL`**, **`COMMITS_URL`** — GitHub URL
+ templates for the `gardenlinux/gardenlinux` repository.
+- **`LIFECYCLE_LINKS`** — Anchor-link map for release lifecycle documentation
+ sections.
+
### `aggregation/fetcher.py`
Repository fetching:
@@ -77,12 +96,87 @@ Content transformation:
- **`parse_frontmatter()`** — Extract metadata from markdown front-matter
- **`fix_broken_project_links()`** — Validate and fix links to project mirrors
+### `aggregation/github_api.py`
+
+GitHub HTTP client for release data:
+
+- **`GitHubAPIError`** — Exception raised on any GitHub API failure (network,
+ non-2xx, rate-limit, or empty first page).
+- **`get_json(url)`** — Fetch a single GitHub API URL and return the parsed
+ JSON response. Uses `GITHUB_TOKEN` when set. Hard-fails on non-2xx status.
+- **`list_repo_releases(owner, repo, per_page=100)`** — Paginate through all
+ releases for a repository and return the full list. Paginates until GitHub
+ returns an empty page (uncapped). Hard-fails on any error or an empty first
+ page.
+
+This module is the single source of truth for GitHub HTTP calls in the
+aggregation package. Future callers should use it rather than adding new
+ad-hoc HTTP requests.
+
+### `aggregation/glrd.py`
+
+GLRD subprocess wrapper:
+
+- **`run_glrd_json(args)`** — Run `glrd` with JSON output and return the parsed
+ data. Returns `None` on any failure (binary not found, non-zero exit, JSON
+ decode error).
+- **`get_active_minor_versions()`** — Return the set of active minor-release
+ version strings from GLRD (e.g. `{"1877.14", "2150.1.0"}`).
+
+### `aggregation/sphinx_builder.py`
+
+Sphinx-to-Markdown builder for repos configured with
+`"structure": "sphinx"` in `repos-config.json`:
+
+- **`build_sphinx_markdown(repo_dir, docs_path, output_dir, target_map=None)`**
+ — Run `python -m sphinx -M markdown` on a fetched repository and copy the
+ resulting Markdown into `output_dir` so the standard Transform/Structure
+ stages can consume it. Injects VitePress frontmatter, carries over
+ hand-written Markdown files that have a `github_target_path` frontmatter
+ field, and strips Sphinx HTML anchors that break VitePress compatibility.
+ Returns `True` on success, `False` on any failure. Requires `sphinx`,
+ `sphinx-markdown-builder`, and any project-specific Sphinx extensions
+ installed in the same Python environment as the aggregator.
+
+### `aggregation/releases.py`
+
+Release-table generation from GLRD data:
+
+- **`generate_release_docs(docs_dir, existing_gh_tags)`** — Read GLRD release
+ data and write per-release documentation pages. Only writes pages for
+ releases whose normalized version string appears in `existing_gh_tags`.
+- **`generate_release_table(releases_data, active_versions,
+existing_gh_tags)`** — Build the release-status table. GLRD-listed rows that
+ carry a `minor` version component are only emitted when their normalized
+ version string appears in `existing_gh_tags`. Major-only rows are always
+ emitted. Skipped rows are logged to `stderr`.
+
+### `aggregation/release_notes.py`
+
+Per-release note page generation:
+
+- **`generate_release_notes_docs(docs_dir, releases)`** — Write one Markdown
+ page per release from the pre-fetched `releases` list. Accepts the release
+ list returned by `github_api.list_repo_releases()` and makes no network
+ calls itself.
+
+### `aggregation/flavor_matrix.py`
+
+Flavor matrix documentation generator:
+
+- **`get_flavor_list(gardenlinux_repo_dir)`** — Parse `flavors.yaml` in the
+ fetched Garden Linux repository and return a per-architecture dict of flavor
+ combinations. Returns `None` on failure.
+- **`generate_flavor_matrix_docs(docs_dir, gardenlinux_repo_dir)`** — Generate
+ the flavor matrix Markdown page by combining `get_flavor_list()` output with
+ feature metadata from the Garden Linux `FeaturesParser`. Appends the
+ generated table to the existing aggregated `reference/flavor-matrix.md` file.
+ Returns `True` on success, `False` on any failure.
+
### `aggregation/structure.py`
Directory operations:
-- **`transform_directory_structure()`** — Restructure docs based on config
- mapping
- **`copy_targeted_docs(source_dir, docs_dir, repo_name, media_dirs=None, root_files=None)`**
— Copy files with `github_target_path` front-matter to specified locations
- Handles nested media dirs (e.g., `tutorials/assets/`) by copying to same
@@ -90,6 +184,11 @@ Directory operations:
- Handles root-level media dirs (e.g., `_static/`) by copying to common
ancestor of targeted files
- Supports scanning root_files for targeted placement
+- **`verify_internal_links(source_dir, docs_dir, repo_name)`** — Verify that
+ all internal relative links in shipped Markdown files resolve to files that
+ were also shipped. Returns the number of broken links found (0 = success).
+ Hard-fails aggregation when any shipped file links to a source-repo file that
+ was not itself shipped.
- **`process_markdown_file()`** — Transform single markdown file (links,
front-matter)
- **`process_all_markdown()`** — Batch process all markdown files in directory
@@ -180,7 +279,7 @@ To add a new transformation:
To add a new structure mapping type:
-1. Update `transform_directory_structure()` in `structure.py`
+1. Add the corresponding logic to `structure.py`
2. Add corresponding structure key handling
3. Update configuration documentation
diff --git a/docs/contributing/documentation/working-locally.md b/docs/contributing/documentation/working-locally.md
index 41232e3..76ee793 100644
--- a/docs/contributing/documentation/working-locally.md
+++ b/docs/contributing/documentation/working-locally.md
@@ -62,6 +62,18 @@ make aggregate
This fetches documentation from the configured repositories at their locked
commit hashes.
+#### GitHub API token
+
+`make aggregate` also fetches every release from `gardenlinux/gardenlinux` via
+the GitHub Releases API to generate per-release note pages and to validate
+GLRD-listed minor releases. Set `GITHUB_TOKEN` before running `make aggregate`
+to prevent rate-limiting:
+
+```bash
+export GITHUB_TOKEN=$(gh auth token)
+make aggregate
+```
+
#### From Local Repositories (Development)
For local development, use `repos-config.local.json` with `file://` URLs:
@@ -175,6 +187,20 @@ python3 --version # Should be 3.x
Check that `repos-config.json` or `repos-config.local.json` is properly
configured. See the [configuration reference](./configuration.md) for details.
+### GitHub API rate limit or fetch error
+
+If `make aggregate` fails with a message like
+`Could not fetch GitHub releases — HTTP 403 …`, the GitHub API rate limit is
+exhausted or the request was rejected. Set `GITHUB_TOKEN` and retry:
+
+```bash
+export GITHUB_TOKEN=$(gh auth token)
+make aggregate
+```
+
+See the [GitHub API token](#github-api-token) note under Step 3 for rate-limit
+details.
+
## Related Topics
diff --git a/src/aggregate.py b/src/aggregate.py
index 736fddd..a7f27ab 100755
--- a/src/aggregate.py
+++ b/src/aggregate.py
@@ -15,6 +15,7 @@
save_config)
from aggregation.structure import verify_internal_links
from aggregation.flavor_matrix import generate_flavor_matrix_docs
+from aggregation.github_api import GitHubAPIError, list_repo_releases
from aggregation.release_notes import generate_release_notes_docs
from aggregation.releases import generate_release_docs
@@ -290,17 +291,36 @@ def main() -> int:
print(f"{'='*60}\n")
generate_flavor_matrix_docs(docs_dir, gardenlinux_temp_dir)
+ # Fetch all GitHub releases up front — used by both release-doc generators.
+ # Hard-fail here rather than inside the generators so the earlier repo-
+ # aggregation results are still printed before we exit non-zero.
+ print(f"\n{'='*60}")
+ print("Fetching GitHub releases...")
+ print(f"{'='*60}\n")
+ try:
+ gh_releases = list_repo_releases("gardenlinux", "gardenlinux")
+ except GitHubAPIError as exc:
+ print(
+ f"\nError: Could not fetch GitHub releases — {exc}\n"
+ "Hint: set GITHUB_TOKEN to avoid rate-limit issues: "
+ "export GITHUB_TOKEN=$(gh auth token)",
+ file=sys.stderr,
+ )
+ return 1
+ existing_gh_tags = {r["tag_name"].lstrip("v") for r in gh_releases}
+ print(f" Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)")
+
# Generate release documentation from GLRD
print(f"\n{'='*60}")
print("Generating release documentation...")
print(f"{'='*60}\n")
- generate_release_docs(docs_dir)
+ generate_release_docs(docs_dir, existing_gh_tags)
- # Generate release notes from GitHub
+ # Generate release notes from pre-fetched GitHub releases
print(f"\n{'='*60}")
- print("Fetching release notes from GitHub...")
+ print("Generating release notes from GitHub...")
print(f"{'='*60}\n")
- generate_release_notes_docs(docs_dir)
+ generate_release_notes_docs(docs_dir, gh_releases)
# Summary
print(f"\n{'='*60}")
diff --git a/src/aggregation/github_api.py b/src/aggregation/github_api.py
new file mode 100644
index 0000000..47e499a
--- /dev/null
+++ b/src/aggregation/github_api.py
@@ -0,0 +1,169 @@
+"""GitHub API HTTP client for docs-ng aggregation.
+
+This module is the single entry point for all GitHub HTTP calls in docs-ng.
+Future callers should use :func:`get_json` or :func:`list_repo_releases` rather
+than writing ad-hoc HTTP requests.
+
+Authentication
+--------------
+Set the ``GITHUB_TOKEN`` environment variable to a personal access token (PAT)
+or an OAuth token. No scopes are required for accessing public repositories.
+When ``GITHUB_TOKEN`` is not set (or is empty after stripping whitespace),
+requests are sent unauthenticated; GitHub applies a rate limit of 60 requests
+per hour for unauthenticated access vs. 5 000 per hour for authenticated access.
+
+Recommended for local development::
+
+ export GITHUB_TOKEN=$(gh auth token)
+
+Failure handling
+----------------
+Any network error, non-2xx HTTP status, rate-limit response (403 with
+``X-RateLimit-Remaining: 0``), or JSON decode error raises
+:class:`GitHubAPIError`. Callers should let this propagate so the make target
+can hard-fail with a non-zero exit code.
+"""
+
+import json
+import os
+import urllib.error
+import urllib.request
+from typing import Any
+
+GITHUB_API_BASE = "https://api.github.com"
+
+
+class GitHubAPIError(Exception):
+ """Raised when any GitHub API request fails.
+
+ The message includes the HTTP status code (when available) and the values
+ of the ``X-RateLimit-Remaining`` and ``X-RateLimit-Reset`` response headers
+ so that CI logs surface actionable information immediately.
+ """
+
+
+def _build_request(url: str) -> urllib.request.Request:
+ """Build a :class:`urllib.request.Request` with standard GitHub API headers.
+
+ Adds ``Authorization: Bearer `` when ``GITHUB_TOKEN`` is set and
+ non-empty.
+ """
+ headers = {
+ "Accept": "application/vnd.github+json",
+ "X-GitHub-Api-Version": "2022-11-28",
+ "User-Agent": "gardenlinux-docs-ng",
+ }
+
+ token = os.environ.get("GITHUB_TOKEN", "").strip()
+ if token:
+ headers["Authorization"] = f"Bearer {token}"
+
+ return urllib.request.Request(url, headers=headers)
+
+
+def get_json(url: str) -> Any:
+ """Perform a GET request to *url* and return the parsed JSON body.
+
+ Args:
+ url: Fully-qualified HTTPS URL of the GitHub API endpoint.
+
+ Returns:
+ Parsed JSON value (usually a :class:`list` or :class:`dict`).
+
+ Raises:
+ GitHubAPIError: On any network error, non-2xx HTTP status, or JSON
+ decode failure. The error message includes rate-limit headers when
+ present.
+ """
+ req = _build_request(url)
+
+ try:
+ with urllib.request.urlopen(req) as response:
+ body = response.read().decode("utf-8")
+ except urllib.error.HTTPError as exc:
+ # Extract rate-limit info from error headers when available.
+ rate_remaining = None
+ rate_reset = None
+ if exc.headers:
+ rate_remaining = exc.headers.get("X-RateLimit-Remaining")
+ rate_reset = exc.headers.get("X-RateLimit-Reset")
+
+ msg = f"GitHub API request failed: HTTP {exc.code} for {url}"
+ if rate_remaining is not None:
+ msg += f" (X-RateLimit-Remaining: {rate_remaining}"
+ if rate_reset is not None:
+ msg += f", X-RateLimit-Reset: {rate_reset}"
+ msg += ")"
+ if rate_remaining == "0":
+ msg += (
+ " — rate limit exhausted; set GITHUB_TOKEN to raise the limit "
+ "(export GITHUB_TOKEN=$(gh auth token))"
+ )
+ raise GitHubAPIError(msg) from exc
+ except urllib.error.URLError as exc:
+ raise GitHubAPIError(
+ f"GitHub API request failed (network error) for {url}: {exc.reason}"
+ ) from exc
+
+ try:
+ return json.loads(body)
+ except json.JSONDecodeError as exc:
+ raise GitHubAPIError(
+ f"GitHub API returned invalid JSON for {url}: {exc}"
+ ) from exc
+
+
+def list_repo_releases(
+ owner: str, repo: str, per_page: int = 100
+) -> list[dict]:
+ """Fetch every release for *owner*/*repo* from the GitHub Releases API.
+
+ Paginates automatically until GitHub returns an empty page. The full list
+ is returned only after all pages have been fetched successfully; a partial
+ list is never returned on error.
+
+ Args:
+ owner: GitHub organisation or user name (e.g. ``"gardenlinux"``).
+ repo: Repository name (e.g. ``"gardenlinux"``).
+ per_page: Number of releases to request per page (max 100).
+
+ Returns:
+ List of release objects as returned by the GitHub Releases API.
+
+ Raises:
+ GitHubAPIError: On any fetch failure *or* when the first page is empty
+ (indicating the repository has no releases, which would break the
+ generated documentation). See module docstring for ``GITHUB_TOKEN``
+ advice.
+ """
+ all_releases: list[dict] = []
+ page = 1
+
+ while True:
+ url = (
+ f"{GITHUB_API_BASE}/repos/{owner}/{repo}/releases"
+ f"?per_page={per_page}&page={page}"
+ )
+ page_data = get_json(url)
+
+ if not isinstance(page_data, list):
+ raise GitHubAPIError(
+ f"GitHub API returned unexpected type for releases page {page}: "
+ f"{type(page_data).__name__}"
+ )
+
+ if len(page_data) == 0:
+ break
+
+ all_releases.extend(page_data)
+ page += 1
+
+ if not all_releases:
+ raise GitHubAPIError(
+ f"GitHub API returned zero releases for {owner}/{repo}. "
+ "If this is unexpected, check that the repository exists and that "
+ "GITHUB_TOKEN (if set) has access to it. "
+ "Set GITHUB_TOKEN to authenticate: export GITHUB_TOKEN=$(gh auth token)"
+ )
+
+ return all_releases
diff --git a/src/aggregation/release_notes.py b/src/aggregation/release_notes.py
index 8af3eb2..2d2881b 100644
--- a/src/aggregation/release_notes.py
+++ b/src/aggregation/release_notes.py
@@ -1,8 +1,6 @@
-"""Fetch and format release notes from GitHub."""
+"""Format release notes from pre-fetched GitHub release data."""
-import json
import re
-import subprocess
import sys
from datetime import datetime
from pathlib import Path
@@ -10,12 +8,10 @@
from .glrd import get_active_minor_versions
from .transformer import cleanup_github_markdown
-GITHUB_API_URL = "https://api.github.com/repos/gardenlinux/gardenlinux/releases"
GITHUB_RELEASES_URL = "https://github.com/gardenlinux/gardenlinux/releases/tag"
GITHUB_COMMITS_URL = "https://github.com/gardenlinux/gardenlinux/commit"
# Configuration
-MAX_RELEASES = 200 # Include up to 200 recent releases
ARCHIVED_DIR = "archived"
@@ -55,39 +51,6 @@ def sort_by_version(releases: list) -> list:
)
-def fetch_github_releases(per_page: int = 100) -> list:
- """Fetch releases from GitHub API using curl with pagination."""
- all_releases = []
- page = 1
- max_pages = (MAX_RELEASES // per_page) + 2 # Fetch enough pages
-
- while page <= max_pages and len(all_releases) < MAX_RELEASES:
- try:
- result = subprocess.run(
- [
- "curl",
- "-s",
- "-L",
- f"{GITHUB_API_URL}?per_page={per_page}&page={page}",
- ],
- capture_output=True,
- text=True,
- check=False,
- )
- if result.returncode == 0:
- page_releases = json.loads(result.stdout)
- if not page_releases:
- break
- all_releases.extend(page_releases)
- else:
- break
- except (json.JSONDecodeError, Exception):
- break
- page += 1
-
- return all_releases[:MAX_RELEASES]
-
-
def format_release_date(date_str: str) -> str:
"""Format ISO date to readable format."""
if not date_str:
@@ -129,8 +92,21 @@ def format_release_entry(release: dict) -> str:
return f"{header}{body.strip()}\n"
-def generate_release_notes_docs(docs_dir: Path) -> bool:
- """Fetch GitHub release notes and generate release-notes as individual files."""
+def generate_release_notes_docs(docs_dir: Path, releases: list[dict]) -> bool:
+ """Generate release-notes documentation files from a pre-fetched release list.
+
+ The caller (``aggregate.py``) is responsible for fetching the full release
+ list via :func:`aggregation.github_api.list_repo_releases` before calling
+ this function. No network requests are made here.
+
+ Args:
+ docs_dir: Root of the VitePress ``docs/`` tree.
+ releases: Full list of GitHub release objects as returned by the GitHub
+ Releases API (already fetched and validated by the caller).
+
+ Returns:
+ ``True`` on success, ``False`` if no releases were provided.
+ """
releases_dir = docs_dir / "reference" / "releases" / "release-notes"
releases_dir.mkdir(parents=True, exist_ok=True)
@@ -143,11 +119,8 @@ def generate_release_notes_docs(docs_dir: Path) -> bool:
md_file.unlink()
print(f" Removed: {md_file.relative_to(docs_dir)}")
- print("Fetching release notes from GitHub...")
- releases = fetch_github_releases()
-
if not releases:
- print("Warning: No releases fetched from GitHub", file=sys.stderr)
+ print("Warning: No releases provided to generate_release_notes_docs", file=sys.stderr)
return False
# Query GLRD to determine release status
@@ -169,9 +142,6 @@ def generate_release_notes_docs(docs_dir: Path) -> bool:
# Sort by semantic version (highest first)
filtered = sort_by_version(filtered)
- # Limit
- filtered = filtered[:MAX_RELEASES]
-
# Generate individual files for each release
release_list = []
for idx, release in enumerate(filtered):
@@ -186,9 +156,9 @@ def generate_release_notes_docs(docs_dir: Path) -> bool:
r"^##\s+" + re.escape(name) + r"$", "# " + name, content, flags=re.MULTILINE
)
- # Determine if this release is archived
- # A release is active ONLY if it's explicitly in the active_versions dict
- # All other releases are archived
+ # Determine if this release is archived.
+ # A release is active ONLY if it's explicitly in the active_versions set.
+ # All other releases are archived.
tag_without_v = tag_name.lstrip("v")
is_archived = tag_without_v not in active_versions
diff --git a/src/aggregation/releases.py b/src/aggregation/releases.py
index 9aa88a3..85bb8e8 100644
--- a/src/aggregation/releases.py
+++ b/src/aggregation/releases.py
@@ -1,4 +1,13 @@
-"""Generate release documentation from GLRD."""
+"""Generate release documentation from GLRD.
+
+The :func:`generate_release_docs` and :func:`generate_release_table` functions
+accept a pre-built ``existing_gh_tags`` set that was fetched from the GitHub
+Releases API by the caller (``aggregate.py``). GLRD rows that carry a
+``minor`` version component are skipped when their normalized version string
+(``{major}.{minor}[.{patch}]``) is absent from that set. Major-only rows are
+always emitted. One warning line per skipped row is written to ``stderr`` so
+that CI logs surface the omission.
+"""
import sys
from pathlib import Path
@@ -133,12 +142,26 @@ def generate_mermaid_gantt(releases_data: dict) -> str:
return gantt
-def generate_release_table(releases_data: dict, active_versions: set[str]) -> str:
+def generate_release_table(
+ releases_data: dict,
+ active_versions: set[str],
+ existing_gh_tags: set[str],
+) -> str:
"""Generate markdown table from GLRD JSON data.
+ GLRD rows that carry a ``minor`` version component are only included when
+ their normalized version string (``{major}.{minor}[.{patch}]``) appears in
+ *existing_gh_tags*. Rows without a matching GitHub release tag are skipped
+ and a warning is written to ``stderr``. Major-only rows are always emitted
+ regardless of *existing_gh_tags*.
+
Args:
- releases_data: Release data from GLRD
- active_versions: Set of active minor version strings
+ releases_data: Release data from GLRD.
+ active_versions: Set of active minor version strings (used for link
+ routing, not for inclusion filtering).
+ existing_gh_tags: Set of GitHub release tag names with leading ``v``
+ stripped (e.g. ``{"1877.14", "2150.1.0"}``). Built from the full
+ list fetched by :func:`aggregation.github_api.list_repo_releases`.
"""
if not releases_data or "releases" not in releases_data:
return "*No releases found*"
@@ -151,6 +174,25 @@ def generate_release_table(releases_data: dict, active_versions: set[str]) -> st
table += "|:--------|:-------|:---------------------|:---------------------|:-------------------|\n"
for release in releases:
+ version_obj = release.get("version", {})
+ has_minor = "minor" in version_obj
+
+ if has_minor:
+ # Build the normalized version string for tag comparison.
+ if "patch" in version_obj:
+ version_str = (
+ f"{version_obj['major']}.{version_obj['minor']}.{version_obj['patch']}"
+ )
+ else:
+ version_str = f"{version_obj['major']}.{version_obj['minor']}"
+
+ if version_str not in existing_gh_tags:
+ print(
+ f"Warning: skipping GLRD version {version_str} — no GitHub release found",
+ file=sys.stderr,
+ )
+ continue
+
_, version_link = format_version(release, active_versions)
commit_link = format_commit(release)
@@ -199,8 +241,20 @@ def append_release_page(
"""
-def generate_release_docs(docs_dir: Path) -> bool:
- """Fetch release data from GLRD and generate release documentation pages."""
+def generate_release_docs(docs_dir: Path, existing_gh_tags: set[str]) -> bool:
+ """Fetch release data from GLRD and generate release documentation pages.
+
+ GLRD minor releases are only included in the generated tables when a
+ matching GitHub release tag exists in *existing_gh_tags* (see
+ :func:`generate_release_table`).
+
+ Args:
+ docs_dir: Root of the VitePress ``docs/`` tree.
+ existing_gh_tags: Set of GitHub release tag names with leading ``v``
+ stripped. Built from the full list fetched by the caller
+ (``aggregate.py``) via
+ :func:`aggregation.github_api.list_repo_releases`.
+ """
releases_dir = docs_dir / "reference" / "releases"
releases_dir.mkdir(parents=True, exist_ok=True)
@@ -220,7 +274,7 @@ def generate_release_docs(docs_dir: Path) -> bool:
)
return False
- active_table = generate_release_table(active_data, active_versions)
+ active_table = generate_release_table(active_data, active_versions, existing_gh_tags)
active_gantt = generate_mermaid_gantt(active_data)
active_timeline = get_timeline_section(active_gantt, "Release Timeline")
@@ -252,7 +306,7 @@ def generate_release_docs(docs_dir: Path) -> bool:
print(f" Updated: {release_path}")
if archived_data is not None:
- archived_table = generate_release_table(archived_data, active_versions)
+ archived_table = generate_release_table(archived_data, active_versions, existing_gh_tags)
archived_gantt = generate_mermaid_gantt(archived_data)
archived_timeline = get_timeline_section(
archived_gantt, "Archived Releases Timeline"
diff --git a/tests/unit/test_github_api.py b/tests/unit/test_github_api.py
new file mode 100644
index 0000000..fccea7b
--- /dev/null
+++ b/tests/unit/test_github_api.py
@@ -0,0 +1,208 @@
+"""Unit tests for aggregation.github_api module."""
+
+import io
+import json
+import os
+import urllib.error
+import urllib.request
+from http.client import HTTPMessage
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from aggregation.github_api import GitHubAPIError, get_json, list_repo_releases
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _make_response(body: bytes, status: int = 200) -> MagicMock:
+ """Return a mock context-manager response that yields *body*."""
+ resp = MagicMock()
+ resp.read.return_value = body
+ resp.__enter__ = lambda s: s
+ resp.__exit__ = MagicMock(return_value=False)
+ return resp
+
+
+def _make_http_error(code: int, headers: dict | None = None) -> urllib.error.HTTPError:
+ """Build a :class:`urllib.error.HTTPError` with optional headers."""
+ msg = HTTPMessage()
+ if headers:
+ for key, val in headers.items():
+ msg[key] = val
+ return urllib.error.HTTPError(
+ url="https://api.github.com/test",
+ code=code,
+ msg=f"HTTP Error {code}",
+ hdrs=msg,
+ fp=io.BytesIO(b""),
+ )
+
+
+# ---------------------------------------------------------------------------
+# get_json
+# ---------------------------------------------------------------------------
+
+class TestGetJson:
+ """Tests for get_json."""
+
+ def test_returns_parsed_json(self):
+ """Successful response returns parsed JSON body."""
+ payload = [{"tag_name": "1.0"}]
+ mock_resp = _make_response(json.dumps(payload).encode())
+
+ with patch("urllib.request.urlopen", return_value=mock_resp):
+ result = get_json("https://api.github.com/repos/x/y/releases")
+
+ assert result == payload
+
+ def test_no_auth_header_without_token(self, monkeypatch):
+ """No Authorization header is set when GITHUB_TOKEN is absent."""
+ monkeypatch.delenv("GITHUB_TOKEN", raising=False)
+
+ captured_req = {}
+
+ def fake_urlopen(req):
+ captured_req["req"] = req
+ return _make_response(b"[]")
+
+ with patch("urllib.request.urlopen", side_effect=fake_urlopen):
+ get_json("https://api.github.com/test")
+
+ assert "Authorization" not in captured_req["req"].headers
+
+ def test_no_auth_header_with_empty_token(self, monkeypatch):
+ """No Authorization header is added when GITHUB_TOKEN is empty / whitespace."""
+ monkeypatch.setenv("GITHUB_TOKEN", " ")
+
+ captured_req = {}
+
+ def fake_urlopen(req):
+ captured_req["req"] = req
+ return _make_response(b"[]")
+
+ with patch("urllib.request.urlopen", side_effect=fake_urlopen):
+ get_json("https://api.github.com/test")
+
+ assert "Authorization" not in captured_req["req"].headers
+
+ def test_auth_header_present_with_token(self, monkeypatch):
+ """Authorization header is set when GITHUB_TOKEN is non-empty."""
+ monkeypatch.setenv("GITHUB_TOKEN", "ghp_testtoken")
+
+ captured_req = {}
+
+ def fake_urlopen(req):
+ captured_req["req"] = req
+ return _make_response(b"[]")
+
+ with patch("urllib.request.urlopen", side_effect=fake_urlopen):
+ get_json("https://api.github.com/test")
+
+ assert captured_req["req"].get_header("Authorization") == "Bearer ghp_testtoken"
+
+ def test_raises_on_http_error(self):
+ """HTTPError is wrapped in GitHubAPIError."""
+ with patch("urllib.request.urlopen", side_effect=_make_http_error(404)):
+ with pytest.raises(GitHubAPIError, match="HTTP 404"):
+ get_json("https://api.github.com/test")
+
+ def test_raises_on_rate_limit_403(self):
+ """403 with zero remaining rate-limit mentions GITHUB_TOKEN in the error."""
+ err = _make_http_error(
+ 403,
+ {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": "1700000000"},
+ )
+ with patch("urllib.request.urlopen", side_effect=err):
+ with pytest.raises(GitHubAPIError, match="GITHUB_TOKEN"):
+ get_json("https://api.github.com/test")
+
+ def test_raises_on_url_error(self):
+ """URLError (network failure) is wrapped in GitHubAPIError."""
+ net_err = urllib.error.URLError(reason="Name or service not known")
+ with patch("urllib.request.urlopen", side_effect=net_err):
+ with pytest.raises(GitHubAPIError, match="network error"):
+ get_json("https://api.github.com/test")
+
+ def test_raises_on_invalid_json(self):
+ """Invalid JSON body raises GitHubAPIError."""
+ with patch("urllib.request.urlopen", return_value=_make_response(b"not-json")):
+ with pytest.raises(GitHubAPIError, match="invalid JSON"):
+ get_json("https://api.github.com/test")
+
+
+# ---------------------------------------------------------------------------
+# list_repo_releases
+# ---------------------------------------------------------------------------
+
+class TestListRepoReleases:
+ """Tests for list_repo_releases."""
+
+ def test_single_page(self):
+ """Single page of releases — returns the page contents."""
+ page1 = [{"tag_name": "1.0"}, {"tag_name": "2.0"}]
+
+ responses = [
+ _make_response(json.dumps(page1).encode()),
+ _make_response(json.dumps([]).encode()),
+ ]
+
+ with patch("urllib.request.urlopen", side_effect=responses):
+ result = list_repo_releases("owner", "repo")
+
+ assert result == page1
+
+ def test_multiple_pages_concatenated(self):
+ """Multiple pages are concatenated into one list."""
+ page1 = [{"tag_name": f"{i}.0"} for i in range(100)]
+ page2 = [{"tag_name": f"{i}.0"} for i in range(100, 150)]
+
+ responses = [
+ _make_response(json.dumps(page1).encode()),
+ _make_response(json.dumps(page2).encode()),
+ _make_response(json.dumps([]).encode()),
+ ]
+
+ with patch("urllib.request.urlopen", side_effect=responses):
+ result = list_repo_releases("owner", "repo")
+
+ assert len(result) == 150
+ assert result == page1 + page2
+
+ def test_stops_on_empty_page(self):
+ """Pagination loop terminates on an empty page, not on IndexError."""
+ page1 = [{"tag_name": "1.0"}]
+
+ responses = [
+ _make_response(json.dumps(page1).encode()),
+ _make_response(b"[]"),
+ ]
+
+ with patch("urllib.request.urlopen", side_effect=responses):
+ result = list_repo_releases("owner", "repo")
+
+ assert len(result) == 1
+
+ def test_raises_on_empty_first_page(self):
+ """Empty first page (repo has no releases) raises GitHubAPIError."""
+ with patch("urllib.request.urlopen", return_value=_make_response(b"[]")):
+ with pytest.raises(GitHubAPIError, match="zero releases"):
+ list_repo_releases("owner", "repo")
+
+ def test_raises_on_fetch_failure(self):
+ """GitHubAPIError from get_json propagates without swallowing."""
+ with patch(
+ "urllib.request.urlopen",
+ side_effect=_make_http_error(500),
+ ):
+ with pytest.raises(GitHubAPIError):
+ list_repo_releases("owner", "repo")
+
+ def test_raises_on_non_list_response(self):
+ """Non-list JSON (e.g. error dict) raises GitHubAPIError."""
+ error_body = json.dumps({"message": "Not Found"}).encode()
+ with patch("urllib.request.urlopen", return_value=_make_response(error_body)):
+ with pytest.raises(GitHubAPIError, match="unexpected type"):
+ list_repo_releases("owner", "repo")
diff --git a/tests/unit/test_release_filter.py b/tests/unit/test_release_filter.py
new file mode 100644
index 0000000..ebad7bb
--- /dev/null
+++ b/tests/unit/test_release_filter.py
@@ -0,0 +1,171 @@
+"""Unit tests for GLRD release-table filtering in aggregation.releases."""
+
+import pytest
+
+from aggregation.releases import generate_release_table
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _make_release(major: int, minor: int | None = None, patch: int | None = None) -> dict:
+ """Build a minimal GLRD release dict."""
+ version: dict = {"major": major}
+ if minor is not None:
+ version["minor"] = minor
+ if patch is not None:
+ version["patch"] = patch
+ return {
+ "version": version,
+ "git": {},
+ "lifecycle": {},
+ }
+
+
+def _make_releases_data(*releases) -> dict:
+ return {"releases": list(releases)}
+
+
+# ---------------------------------------------------------------------------
+# Filtering: minor rows
+# ---------------------------------------------------------------------------
+
+class TestGenerateReleaseTableFiltering:
+ """generate_release_table skips minor rows absent from existing_gh_tags."""
+
+ def test_minor_row_included_when_tag_present(self):
+ """A minor release present in existing_gh_tags appears in the table."""
+ data = _make_releases_data(_make_release(1877, 14))
+ table = generate_release_table(data, set(), existing_gh_tags={"1877.14"})
+ assert "1877.14" in table
+
+ def test_minor_row_skipped_when_tag_absent(self):
+ """A minor release absent from existing_gh_tags is dropped."""
+ data = _make_releases_data(_make_release(1877, 14))
+ table = generate_release_table(data, set(), existing_gh_tags=set())
+ assert "1877.14" not in table
+ # Table should be effectively empty (header rows only)
+ data_rows = [
+ line for line in table.splitlines()
+ if line.startswith("|") and not line.startswith("|:") and "Version" not in line
+ ]
+ assert data_rows == []
+
+ def test_major_row_always_included(self):
+ """A major-only row is emitted regardless of existing_gh_tags."""
+ data = _make_releases_data(_make_release(1877))
+ table = generate_release_table(data, set(), existing_gh_tags=set())
+ assert "1877" in table
+
+ def test_mixed_major_and_minor(self):
+ """Major rows pass through; unmatched minor rows are dropped."""
+ data = _make_releases_data(
+ _make_release(2150), # major-only — always kept
+ _make_release(2150, 1, 0), # minor — tag present
+ _make_release(2150, 2, 0), # minor — tag absent
+ )
+ table = generate_release_table(
+ data,
+ active_versions={"2150.1.0"},
+ existing_gh_tags={"2150.1.0"},
+ )
+ assert "2150.1.0" in table
+ assert "2150.2.0" not in table
+ # Major row 2150 is present (as plain text, no dot-separated minor)
+ assert "2150" in table
+
+ def test_v_prefix_stripped_from_gh_tag(self):
+ """Tags stored with a leading 'v' in existing_gh_tags are still matched."""
+ # The caller strips 'v' before building the set, but simulate the raw
+ # comparison working by stripping manually (documents the contract).
+ raw_tags = {"v1877.14", "v2150.1.0"}
+ normalized = {t.lstrip("v") for t in raw_tags}
+
+ data = _make_releases_data(_make_release(1877, 14))
+ table = generate_release_table(data, set(), existing_gh_tags=normalized)
+ assert "1877.14" in table
+
+ def test_patch_version_compared_correctly(self):
+ """Minor+patch version strings are matched against existing_gh_tags."""
+ data = _make_releases_data(_make_release(2150, 1, 0))
+ table = generate_release_table(data, set(), existing_gh_tags={"2150.1.0"})
+ assert "2150.1.0" in table
+
+ def test_patch_version_absent_skipped(self):
+ """Minor+patch version absent from existing_gh_tags is skipped."""
+ data = _make_releases_data(_make_release(2150, 1, 0))
+ table = generate_release_table(data, set(), existing_gh_tags={"2150.1"})
+ # "2150.1.0" != "2150.1" — strict equality after normalization
+ assert "2150.1.0" not in table
+
+ def test_warning_written_to_stderr_for_skipped(self, capsys):
+ """A warning line is written to stderr for each skipped minor release."""
+ data = _make_releases_data(_make_release(1877, 99))
+ generate_release_table(data, set(), existing_gh_tags=set())
+ captured = capsys.readouterr()
+ assert "1877.99" in captured.err
+ assert "Warning" in captured.err
+
+ def test_no_warning_for_included_minor(self, capsys):
+ """No warning is written when the minor release tag is present."""
+ data = _make_releases_data(_make_release(1877, 14))
+ generate_release_table(data, set(), existing_gh_tags={"1877.14"})
+ captured = capsys.readouterr()
+ assert "1877.14" not in captured.err
+
+ def test_empty_releases_data(self):
+ """Empty releases dict returns the 'no releases' sentinel."""
+ table = generate_release_table({}, set(), existing_gh_tags=set())
+ assert "*No releases found*" in table
+
+ def test_empty_releases_list(self):
+ """Empty releases list returns the 'no releases' sentinel."""
+ table = generate_release_table({"releases": []}, set(), existing_gh_tags=set())
+ assert "*No releases found*" in table
+
+ def test_glrd_release_exists_but_no_github_release(self):
+ """GLRD lists a minor release that has no corresponding GitHub release tag.
+
+ This is the primary guard this filtering was designed for: a GLRD entry
+ whose tag was never pushed to GitHub (e.g. a planned but unpublished
+ release). The row must be dropped from the generated table and a
+ warning must appear on stderr.
+ """
+ # GitHub has released 2150.0.0 and 2150.1.0 but NOT 2150.2.0.
+ # GLRD, however, already lists 2150.2.0 (e.g. an upcoming release).
+ existing_gh_tags = {"2150.0.0", "2150.1.0"}
+ data = _make_releases_data(
+ _make_release(2150, 0, 0),
+ _make_release(2150, 1, 0),
+ _make_release(2150, 2, 0), # in GLRD, but NO matching GitHub release
+ )
+ table = generate_release_table(
+ data,
+ active_versions={"2150.1.0"},
+ existing_gh_tags=existing_gh_tags,
+ )
+
+ # Releases that DO have GitHub tags are included.
+ assert "2150.0.0" in table
+ assert "2150.1.0" in table
+ # The GLRD-only entry is absent.
+ assert "2150.2.0" not in table
+
+ def test_glrd_release_exists_but_no_github_release_emits_warning(self, capsys):
+ """A warning naming the missing version is written to stderr."""
+ existing_gh_tags = {"2150.0.0"}
+ data = _make_releases_data(
+ _make_release(2150, 0, 0),
+ _make_release(2150, 1, 0), # GLRD only — no GitHub tag
+ )
+ generate_release_table(
+ data,
+ active_versions=set(),
+ existing_gh_tags=existing_gh_tags,
+ )
+ captured = capsys.readouterr()
+ assert "2150.1.0" in captured.err
+ assert "Warning" in captured.err
+ # The present release must NOT trigger a warning.
+ assert "2150.0.0" not in captured.err