diff --git a/docs/contributing/documentation/aggregation-architecture.md b/docs/contributing/documentation/aggregation-architecture.md index b9272a5..9df76c9 100644 --- a/docs/contributing/documentation/aggregation-architecture.md +++ b/docs/contributing/documentation/aggregation-architecture.md @@ -103,7 +103,33 @@ flowchart TD - Handles root files separately from docs directory - Provides commit hash for reproducible builds -### 2. Transform Stage (`transformer.py`) +### 2. GitHub HTTP client (`github_api.py`) + +**Purpose:** Fetch GitHub release data for use by the release generators. + +Uses `GITHUB_TOKEN` when set. Hard-fails on any error (network, non-2xx status, +rate-limit exhaustion, or empty first page) so release docs never ship in a +silently degraded state. + +### 3. GLRD subprocess wrapper (`glrd.py`) + +**Purpose:** Query active minor-release versions and release metadata from the +`glrd` command-line tool. The data drives the release-table generator. + +Returns `None` on any failure rather than raising exceptions. + +### 4. Sphinx Markdown builder (`sphinx_builder.py`) + +**Purpose:** Produce plain Markdown output from Sphinx-based documentation so +it can be consumed by VitePress through the normal aggregation pipeline. + +Invokes `python -m sphinx -M markdown` on a fetched repository and copies the +resulting Markdown into the aggregation output directory. Called only for repos +configured with `"structure": "sphinx"` in `repos-config.json`. Requires +`sphinx`, `sphinx-markdown-builder`, and any project-specific Sphinx extensions +installed in the same Python environment as the aggregator. + +### 5. Transform Stage (`transformer.py`) **Purpose:** Modify content to work in the aggregated site @@ -121,7 +147,7 @@ flowchart TD - Quote YAML values safely - Preserve existing metadata -### 3. Structure Stage (`structure.py`) +### 6. Structure Stage (`structure.py`) **Purpose:** Organize documentation into the final directory structure @@ -135,6 +161,28 @@ flowchart TD 3. **Media Copying:** Discover and copy media directories 4. **Markdown Processing:** Apply front-matter fixes to all copied files +### 7. Release-table generator (`releases.py`) + +**Purpose:** Build the release-status table from GLRD data, filtered against +fetched GitHub tags. + +GLRD-listed rows that carry a `minor` version component are only emitted when +their normalized version string (`{major}.{minor}[.{patch}]`, leading `v` +stripped) appears in the fetched GitHub tag set. Major-only rows are always +emitted. Each skipped row logs a one-line warning to `stderr`. + +### 8. Per-release notes generator (`release_notes.py`) + +**Purpose:** Write one Markdown page per pre-fetched GitHub release. Makes no +network calls; accepts the release list returned by +`github_api.list_repo_releases()`. + +### 9. Flavor matrix generator (`flavor_matrix.py`) + +**Purpose:** Generate flavor matrix documentation from `flavors.yaml` and +feature metadata. Parses `flavors.yaml` directly using the Garden Linux +`FlavorsParser` and `FeaturesParser` classes. + ## Key Mechanisms ### Targeted Documentation @@ -262,7 +310,7 @@ Temp Directory Docs Output ### Structure Stage - **Targeted copy:** O(n) where n = files with github_target_path -- **Link verification:** O(n * l) where l = avg links per file +- **Link verification:** O(n \* l) where l = avg links per file - **Media copy:** O(m) where m = media files ### Overall @@ -291,6 +339,13 @@ Temp Directory Docs Output - Conflicting file paths → Error with clear message - Media directory not found → Log warning, continue +### GitHub API Failures + +Any GitHub API failure during release pre-fetch (network, non-2xx status, +rate-limit exhaustion, or empty result) is fatal. This prevents shipping +silently degraded release documentation. See +[GitHub API token](./working-locally.md#github-api-token) for token setup. + ## Related Topics diff --git a/docs/contributing/documentation/ci-workflows-reference.md b/docs/contributing/documentation/ci-workflows-reference.md index 696022d..599ec37 100644 --- a/docs/contributing/documentation/ci-workflows-reference.md +++ b/docs/contributing/documentation/ci-workflows-reference.md @@ -19,11 +19,11 @@ related_topics: ## 1. `docs-check.yml` — Per-Repo Caller Workflow -| | | -|---|---| -| **File** | `.github/workflows/docs-check.yml` | -| **Lives in** | Each aggregated repository | -| **Type** | Caller workflow — invokes the reusable `docs-checks.yml` from `docs-ng` | +| | | +| ------------ | ----------------------------------------------------------------------- | +| **File** | `.github/workflows/docs-check.yml` | +| **Lives in** | Each aggregated repository | +| **Type** | Caller workflow — invokes the reusable `docs-checks.yml` from `docs-ng` | ### Triggers @@ -33,12 +33,12 @@ on: branches: [main, docs-ng] types: [opened, synchronize, reopened, closed] paths: - - 'docs/**' - - 'features/*/README.md' - - 'features/*/info.yaml' - - 'flavors.yaml' - - 'CONTRIBUTING.md' - - 'SECURITY.md' + - "docs/**" + - "features/*/README.md" + - "features/*/info.yaml" + - "flavors.yaml" + - "CONTRIBUTING.md" + - "SECURITY.md" push: branches: [main, docs-ng] paths: @@ -51,18 +51,18 @@ can detect merged PRs. ### Jobs -| Job | Condition | Purpose | -|-----|-----------|---------| -| `docs-checks` | Skipped when `action == 'closed'` | Calls the reusable quality-check workflow in `docs-ng` with override inputs | +| Job | Condition | Purpose | +| ---------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | +| `docs-checks` | Skipped when `action == 'closed'` | Calls the reusable quality-check workflow in `docs-ng` with override inputs | | `notify-docs-ng` | Runs after `docs-checks` succeeds **or** when a PR is merged (always-conditional) | Generates a GitHub App token and sends a `repository_dispatch` to `docs-ng` | ### Inputs Passed to Reusable Workflow -| Input | Value | -|-------|-------| -| `override-repo` | Hard-coded repository name (e.g., `gardenlinux`) | -| `override-ref` | `github.head_ref` (PR branch) or `github.ref_name` (push) | -| `override-commit` | `pull_request.head.sha` or `github.sha` | +| Input | Value | +| ----------------- | --------------------------------------------------------- | +| `override-repo` | Hard-coded repository name (e.g., `gardenlinux`) | +| `override-ref` | `github.head_ref` (PR branch) or `github.ref_name` (push) | +| `override-commit` | `pull_request.head.sha` or `github.sha` | ### Dispatch Payload Sent to `docs-ng` @@ -78,9 +78,9 @@ can detect merged PRs. ### Required Secrets -| Secret | Purpose | -|--------|---------| -| `DOCS_BOT_APP_ID` | GitHub App identifier for `gardenlinux-docs-bot` | +| Secret | Purpose | +| ---------------------- | --------------------------------------------------------- | +| `DOCS_BOT_APP_ID` | GitHub App identifier for `gardenlinux-docs-bot` | | `DOCS_BOT_PRIVATE_KEY` | Private key used to sign the JWT for App token generation | ### Outputs / Side-Effects @@ -110,20 +110,19 @@ flowchart TD ### Common Failure Causes -| Symptom | Likely cause | -|---------|--------------| -| `docs-checks` job fails | Broken links, spelling errors, or woke violations in the documentation change | +| Symptom | Likely cause | +| -------------------------- | -------------------------------------------------------------------------------------------------- | +| `docs-checks` job fails | Broken links, spelling errors, or woke violations in the documentation change | | `notify-docs-ng` job fails | `DOCS_BOT_APP_ID` or `DOCS_BOT_PRIVATE_KEY` secret missing/invalid; App not installed on `docs-ng` | -| Workflow not triggered | Changed files do not match the `paths:` filter | - +| Workflow not triggered | Changed files do not match the `paths:` filter | ## 2. `docs-checks.yml` — Reusable Quality Checks Workflow -| | | -|---|---| -| **File** | `.github/workflows/docs-checks.yml` | -| **Lives in** | `gardenlinux/docs-ng` | -| **Type** | Reusable workflow (`workflow_call`) + own PR/push trigger | +| | | +| ------------ | --------------------------------------------------------- | +| **File** | `.github/workflows/docs-checks.yml` | +| **Lives in** | `gardenlinux/docs-ng` | +| **Type** | Reusable workflow (`workflow_call`) + own PR/push trigger | ### Triggers @@ -132,15 +131,15 @@ on: workflow_call: inputs: override-repo: - description: 'Repository name to override (e.g., gardenlinux)' + description: "Repository name to override (e.g., gardenlinux)" required: false type: string override-ref: - description: 'Git ref to use for the overridden repo' + description: "Git ref to use for the overridden repo" required: false type: string override-commit: - description: 'Git commit SHA to use for the overridden repo' + description: "Git commit SHA to use for the overridden repo" required: false type: string pull_request: @@ -155,11 +154,11 @@ own documentation (with no overrides — all repos use their pinned commits). ### Inputs -| Input | Required | Description | -|-------|----------|-------------| -| `override-repo` | no | Repository name in `repos-config.json` to override | -| `override-ref` | no | Git ref (branch) to check out for the overridden repo | -| `override-commit` | no | Exact commit SHA to aggregate from | +| Input | Required | Description | +| ----------------- | -------- | ----------------------------------------------------- | +| `override-repo` | no | Repository name in `repos-config.json` to override | +| `override-ref` | no | Git ref (branch) to check out for the overridden repo | +| `override-commit` | no | Exact commit SHA to aggregate from | When all three inputs are provided, `aggregate.py` substitutes the pinned commit in `repos-config.json` with the specified commit, effectively testing @@ -167,18 +166,18 @@ the PR's documentation against the rest of the published site. ### Jobs -| Job | Depends on | Runner | Purpose | -|-----|-----------|--------|---------| -| `aggregate` | — | `ubuntu-24.04` | Runs `aggregate.py` with override arguments; uploads `docs/` as artifact | -| `linkcheck` | `aggregate` | `ubuntu-24.04` | Runs lychee link checker on the aggregated `docs/**/*.md` | -| `spelling` | `aggregate` | `ubuntu-24.04` | Runs `make spelling` (codespell) | -| `woke` | `aggregate` | `ubuntu-24.04` | Runs `make woke` (inclusive-language checker) | +| Job | Depends on | Runner | Purpose | +| ----------- | ----------- | -------------- | ------------------------------------------------------------------------ | +| `aggregate` | — | `ubuntu-24.04` | Runs `aggregate.py` with override arguments; uploads `docs/` as artifact | +| `linkcheck` | `aggregate` | `ubuntu-24.04` | Runs lychee link checker on the aggregated `docs/**/*.md` | +| `spelling` | `aggregate` | `ubuntu-24.04` | Runs `make spelling` (codespell) | +| `woke` | `aggregate` | `ubuntu-24.04` | Runs `make woke` (inclusive-language checker) | ### Artifacts -| Artifact | Contents | Retention | -|----------|----------|-----------| -| `aggregated-docs` | Full `docs/` directory after aggregation | 1 day | +| Artifact | Contents | Retention | +| ----------------- | ---------------------------------------- | --------- | +| `aggregated-docs` | Full `docs/` directory after aggregation | 1 day | The artifact is consumed by the three parallel lint jobs (`linkcheck`, `spelling`, `woke`) to avoid re-running aggregation. @@ -189,6 +188,10 @@ No secrets are required by this workflow itself (it only reads public repositories). When called as a reusable workflow, it inherits the caller's `GITHUB_TOKEN` permissions (read-only is sufficient). +:::tip `GITHUB_TOKEN` +Setting the `GITHUB_TOKEN` environment variable is recommended to prevent rate limiting. +::: + ### Job Graph ```mermaid @@ -207,20 +210,20 @@ flowchart TD ### Common Failure Causes -| Symptom | Likely cause | -|---------|--------------| +| Symptom | Likely cause | +| --------------------- | ------------------------------------------------------------------------------------------------------------ | | `aggregate` job fails | Invalid `repos-config.json` entry for the overridden repo; commit SHA not reachable; Python dependency issue | -| `linkcheck` job fails | Broken internal or external links in Markdown files; check lychee output for specific URLs | -| `spelling` job fails | Unknown words — add them to the project's codespell dictionary (`.codespell/project-words.txt`) | -| `woke` job fails | Non-inclusive language detected — see the woke rule file (`.woke.yml`) for alternatives | +| `linkcheck` job fails | Broken internal or external links in Markdown files; check lychee output for specific URLs | +| `spelling` job fails | Unknown words — add them to the project's codespell dictionary (`.codespell/project-words.txt`) | +| `woke` job fails | Non-inclusive language detected — see the woke rule file (`.woke.yml`) for alternatives | ## 3. `docs-pr.yml` — Automated Docs PR Workflow -| | | -|---|---| -| **File** | `.github/workflows/docs-pr.yml` | -| **Lives in** | `gardenlinux/docs-ng` | -| **Type** | `repository_dispatch` consumer + `workflow_dispatch` (manual) | +| | | +| ------------ | ------------------------------------------------------------- | +| **File** | `.github/workflows/docs-pr.yml` | +| **Lives in** | `gardenlinux/docs-ng` | +| **Type** | `repository_dispatch` consumer + `workflow_dispatch` (manual) | ### Triggers @@ -230,11 +233,11 @@ on: types: [docs-pr] workflow_dispatch: inputs: - repo: # required, string - pr_number: # required, string - commit_sha: # required, string - ref: # required, string - event: # required, choice: [pr_success, merged] + repo: # required, string + pr_number: # required, string + commit_sha: # required, string + ref: # required, string + event: # required, choice: [pr_success, merged] ``` The workflow is normally triggered by `repository_dispatch` from a source @@ -245,18 +248,18 @@ repo's `docs-check.yml`. It can also be triggered manually via The `client_payload` (or `inputs` for manual dispatch) must contain: -| Field | Type | Description | -|-------|------|-------------| -| `repo` | string | Repository name as it appears in `repos-config.json` (e.g., `gardenlinux`) | -| `pr_number` | string | PR number in the source repository | -| `commit_sha` | string | Commit SHA to pin in `repos-config.json` | -| `ref` | string | Branch name — feature branch for open PRs, base branch (e.g., `main`) for merged PRs | -| `event` | string | Either `pr_success` (PR is open, checks passed) or `merged` (PR was merged) | +| Field | Type | Description | +| ------------ | ------ | ------------------------------------------------------------------------------------ | +| `repo` | string | Repository name as it appears in `repos-config.json` (e.g., `gardenlinux`) | +| `pr_number` | string | PR number in the source repository | +| `commit_sha` | string | Commit SHA to pin in `repos-config.json` | +| `ref` | string | Branch name — feature branch for open PRs, base branch (e.g., `main`) for merged PRs | +| `event` | string | Either `pr_success` (PR is open, checks passed) or `merged` (PR was merged) | ### Jobs -| Job | Runner | Purpose | -|-----|--------|---------| +| Job | Runner | Purpose | +| ----------- | -------------- | --------------------------------------------------------------------------------------------------------- | | `create-pr` | `ubuntu-24.04` | Creates/updates a branch, modifies `repos-config.json`, opens/updates a PR, and comments on the source PR | ### Job Steps (in order) @@ -280,19 +283,19 @@ The `client_payload` (or `inputs` for manual dispatch) must contain: ### Required Secrets -| Secret | Purpose | -|--------|---------| -| `DOCS_BOT_APP_ID` | GitHub App identifier (configured at org level in `docs-ng`) | -| `DOCS_BOT_PRIVATE_KEY` | Private key for JWT token generation | +| Secret | Purpose | +| ---------------------- | ------------------------------------------------------------ | +| `DOCS_BOT_APP_ID` | GitHub App identifier (configured at org level in `docs-ng`) | +| `DOCS_BOT_PRIVATE_KEY` | Private key for JWT token generation | ### Outputs / Side-Effects -| Side-effect | Details | -|-------------|---------| -| Branch `auto//` | Created or force-updated in `docs-ng` | -| Draft PR in `docs-ng` | Title: `docs(): Update commit lock from PR #` | -| PR transitioned to ready-for-review | When `event == merged` | -| Comment on source PR | Contains docs-ng PR URL and Netlify preview link | +| Side-effect | Details | +| ----------------------------------- | ------------------------------------------------------ | +| Branch `auto//` | Created or force-updated in `docs-ng` | +| Draft PR in `docs-ng` | Title: `docs(): Update commit lock from PR #` | +| PR transitioned to ready-for-review | When `event == merged` | +| Comment on source PR | Contains docs-ng PR URL and Netlify preview link | ### Branch Naming Convention @@ -327,20 +330,20 @@ flowchart TD ### Common Failure Causes -| Symptom | Likely cause | -|---------|--------------| -| Job fails at "Generate GitHub App token" | `DOCS_BOT_APP_ID` or `DOCS_BOT_PRIVATE_KEY` not set in `docs-ng` repository secrets | -| "Repository not found in repos-config.json" | The dispatching repo's name does not match any `name` field in `repos-config.json` | -| Force-push fails | Branch protection rules on `docs-ng` blocking the bot (auto branches should not be protected) | -| Comment not posted | App token lacks write access to the source repo's issues/PRs | +| Symptom | Likely cause | +| ------------------------------------------- | --------------------------------------------------------------------------------------------- | +| Job fails at "Generate GitHub App token" | `DOCS_BOT_APP_ID` or `DOCS_BOT_PRIVATE_KEY` not set in `docs-ng` repository secrets | +| "Repository not found in repos-config.json" | The dispatching repo's name does not match any `name` field in `repos-config.json` | +| Force-push fails | Branch protection rules on `docs-ng` blocking the bot (auto branches should not be protected) | +| Comment not posted | App token lacks write access to the source repo's issues/PRs | ## 4. `check-pr-main.yml` — `repos-config.json` Validator -| | | -|---|---| -| **File** | `.github/workflows/check-pr-main.yml` | -| **Lives in** | `gardenlinux/docs-ng` | -| **Type** | PR validator (status check) | +| | | +| ------------ | ------------------------------------- | +| **File** | `.github/workflows/check-pr-main.yml` | +| **Lives in** | `gardenlinux/docs-ng` | +| **Type** | PR validator (status check) | ### Triggers @@ -373,8 +376,8 @@ check fails, blocking the merge. ### Jobs -| Job | Runner | Purpose | -|-----|--------|---------| +| Job | Runner | Purpose | +| ---------- | -------------- | -------------------------------------------------------------------------- | | `validate` | `ubuntu-24.04` | Reads `repos-config.json`, iterates over repos, validates each `ref` field | ### Required Secrets / Permissions @@ -408,11 +411,11 @@ flowchart TD ### Common Failure Causes -| Symptom | Likely cause | -|---------|--------------| -| Check fails with "non-main branch references" | The automated PR was created from an open (not yet merged) source PR — the ref still points at the feature branch | -| Check fails after source PR was merged | Race condition — the `docs-pr.yml` workflow has not yet run to update the ref. Wait for the "merged" dispatch to complete and force-push the updated branch | -| False positive on `docs-ng` ref | The `docs-ng` ref is in the allowed set — if validation fails, check the exact spelling in `repos-config.json` | +| Symptom | Likely cause | +| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Check fails with "non-main branch references" | The automated PR was created from an open (not yet merged) source PR — the ref still points at the feature branch | +| Check fails after source PR was merged | Race condition — the `docs-pr.yml` workflow has not yet run to update the ref. Wait for the "merged" dispatch to complete and force-push the updated branch | +| False positive on `docs-ng` ref | The `docs-ng` ref is in the allowed set — if validation fails, check the exact spelling in `repos-config.json` | ## Related Topics diff --git a/docs/contributing/documentation/technical.md b/docs/contributing/documentation/technical.md index 3fd8bc7..e488987 100644 --- a/docs/contributing/documentation/technical.md +++ b/docs/contributing/documentation/technical.md @@ -27,11 +27,18 @@ src/ ├── migration_tracker.py # Standalone utility └── aggregation/ # Core package ├── __init__.py - ├── models.py # Data classes ├── config.py # Config I/O + ├── constants.py # Shared constants (Mermaid theme, GitHub URL templates) ├── fetcher.py # Git + local fetch - ├── transformer.py # Content transforms - └── structure.py # Directory transforms + ├── flavor_matrix.py # Flavor matrix docs from flavors.yaml + ├── github_api.py # GitHub HTTP client (releases fetch, auth) + ├── glrd.py # GLRD subprocess wrapper (active versions, metadata) + ├── models.py # Data classes + ├── release_notes.py # Per-release note pages from GH data + ├── releases.py # Release tables from GLRD (filtered by GH tags) + ├── sphinx_builder.py # Sphinx-to-Markdown builder for sphinx-structured repos + ├── structure.py # Directory transforms + └── transformer.py # Content transforms ``` ## Module Reference @@ -50,6 +57,18 @@ Configuration file handling: - **`load_config()`** — Parse repos-config.json - **`save_config()`** — Write updated config (commit locks) +### `aggregation/constants.py` + +Shared constants for release-doc generation. Contains no functions; exposes +only module-level constants: + +- **`GANTT_THEME`** — Mermaid Gantt chart theme variables using the Garden + Linux color palette. +- **`GITHUB_BASE_URL`**, **`RELEASES_TAG_URL`**, **`COMMITS_URL`** — GitHub URL + templates for the `gardenlinux/gardenlinux` repository. +- **`LIFECYCLE_LINKS`** — Anchor-link map for release lifecycle documentation + sections. + ### `aggregation/fetcher.py` Repository fetching: @@ -77,12 +96,87 @@ Content transformation: - **`parse_frontmatter()`** — Extract metadata from markdown front-matter - **`fix_broken_project_links()`** — Validate and fix links to project mirrors +### `aggregation/github_api.py` + +GitHub HTTP client for release data: + +- **`GitHubAPIError`** — Exception raised on any GitHub API failure (network, + non-2xx, rate-limit, or empty first page). +- **`get_json(url)`** — Fetch a single GitHub API URL and return the parsed + JSON response. Uses `GITHUB_TOKEN` when set. Hard-fails on non-2xx status. +- **`list_repo_releases(owner, repo, per_page=100)`** — Paginate through all + releases for a repository and return the full list. Paginates until GitHub + returns an empty page (uncapped). Hard-fails on any error or an empty first + page. + +This module is the single source of truth for GitHub HTTP calls in the +aggregation package. Future callers should use it rather than adding new +ad-hoc HTTP requests. + +### `aggregation/glrd.py` + +GLRD subprocess wrapper: + +- **`run_glrd_json(args)`** — Run `glrd` with JSON output and return the parsed + data. Returns `None` on any failure (binary not found, non-zero exit, JSON + decode error). +- **`get_active_minor_versions()`** — Return the set of active minor-release + version strings from GLRD (e.g. `{"1877.14", "2150.1.0"}`). + +### `aggregation/sphinx_builder.py` + +Sphinx-to-Markdown builder for repos configured with +`"structure": "sphinx"` in `repos-config.json`: + +- **`build_sphinx_markdown(repo_dir, docs_path, output_dir, target_map=None)`** + — Run `python -m sphinx -M markdown` on a fetched repository and copy the + resulting Markdown into `output_dir` so the standard Transform/Structure + stages can consume it. Injects VitePress frontmatter, carries over + hand-written Markdown files that have a `github_target_path` frontmatter + field, and strips Sphinx HTML anchors that break VitePress compatibility. + Returns `True` on success, `False` on any failure. Requires `sphinx`, + `sphinx-markdown-builder`, and any project-specific Sphinx extensions + installed in the same Python environment as the aggregator. + +### `aggregation/releases.py` + +Release-table generation from GLRD data: + +- **`generate_release_docs(docs_dir, existing_gh_tags)`** — Read GLRD release + data and write per-release documentation pages. Only writes pages for + releases whose normalized version string appears in `existing_gh_tags`. +- **`generate_release_table(releases_data, active_versions, +existing_gh_tags)`** — Build the release-status table. GLRD-listed rows that + carry a `minor` version component are only emitted when their normalized + version string appears in `existing_gh_tags`. Major-only rows are always + emitted. Skipped rows are logged to `stderr`. + +### `aggregation/release_notes.py` + +Per-release note page generation: + +- **`generate_release_notes_docs(docs_dir, releases)`** — Write one Markdown + page per release from the pre-fetched `releases` list. Accepts the release + list returned by `github_api.list_repo_releases()` and makes no network + calls itself. + +### `aggregation/flavor_matrix.py` + +Flavor matrix documentation generator: + +- **`get_flavor_list(gardenlinux_repo_dir)`** — Parse `flavors.yaml` in the + fetched Garden Linux repository and return a per-architecture dict of flavor + combinations. Returns `None` on failure. +- **`generate_flavor_matrix_docs(docs_dir, gardenlinux_repo_dir)`** — Generate + the flavor matrix Markdown page by combining `get_flavor_list()` output with + feature metadata from the Garden Linux `FeaturesParser`. Appends the + generated table to the existing aggregated `reference/flavor-matrix.md` file. + Returns `True` on success, `False` on any failure. + ### `aggregation/structure.py` Directory operations: -- **`transform_directory_structure()`** — Restructure docs based on config - mapping - **`copy_targeted_docs(source_dir, docs_dir, repo_name, media_dirs=None, root_files=None)`** — Copy files with `github_target_path` front-matter to specified locations - Handles nested media dirs (e.g., `tutorials/assets/`) by copying to same @@ -90,6 +184,11 @@ Directory operations: - Handles root-level media dirs (e.g., `_static/`) by copying to common ancestor of targeted files - Supports scanning root_files for targeted placement +- **`verify_internal_links(source_dir, docs_dir, repo_name)`** — Verify that + all internal relative links in shipped Markdown files resolve to files that + were also shipped. Returns the number of broken links found (0 = success). + Hard-fails aggregation when any shipped file links to a source-repo file that + was not itself shipped. - **`process_markdown_file()`** — Transform single markdown file (links, front-matter) - **`process_all_markdown()`** — Batch process all markdown files in directory @@ -180,7 +279,7 @@ To add a new transformation: To add a new structure mapping type: -1. Update `transform_directory_structure()` in `structure.py` +1. Add the corresponding logic to `structure.py` 2. Add corresponding structure key handling 3. Update configuration documentation diff --git a/docs/contributing/documentation/working-locally.md b/docs/contributing/documentation/working-locally.md index 41232e3..76ee793 100644 --- a/docs/contributing/documentation/working-locally.md +++ b/docs/contributing/documentation/working-locally.md @@ -62,6 +62,18 @@ make aggregate This fetches documentation from the configured repositories at their locked commit hashes. +#### GitHub API token + +`make aggregate` also fetches every release from `gardenlinux/gardenlinux` via +the GitHub Releases API to generate per-release note pages and to validate +GLRD-listed minor releases. Set `GITHUB_TOKEN` before running `make aggregate` +to prevent rate-limiting: + +```bash +export GITHUB_TOKEN=$(gh auth token) +make aggregate +``` + #### From Local Repositories (Development) For local development, use `repos-config.local.json` with `file://` URLs: @@ -175,6 +187,20 @@ python3 --version # Should be 3.x Check that `repos-config.json` or `repos-config.local.json` is properly configured. See the [configuration reference](./configuration.md) for details. +### GitHub API rate limit or fetch error + +If `make aggregate` fails with a message like +`Could not fetch GitHub releases — HTTP 403 …`, the GitHub API rate limit is +exhausted or the request was rejected. Set `GITHUB_TOKEN` and retry: + +```bash +export GITHUB_TOKEN=$(gh auth token) +make aggregate +``` + +See the [GitHub API token](#github-api-token) note under Step 3 for rate-limit +details. + ## Related Topics diff --git a/src/aggregate.py b/src/aggregate.py index 736fddd..a7f27ab 100755 --- a/src/aggregate.py +++ b/src/aggregate.py @@ -15,6 +15,7 @@ save_config) from aggregation.structure import verify_internal_links from aggregation.flavor_matrix import generate_flavor_matrix_docs +from aggregation.github_api import GitHubAPIError, list_repo_releases from aggregation.release_notes import generate_release_notes_docs from aggregation.releases import generate_release_docs @@ -290,17 +291,36 @@ def main() -> int: print(f"{'='*60}\n") generate_flavor_matrix_docs(docs_dir, gardenlinux_temp_dir) + # Fetch all GitHub releases up front — used by both release-doc generators. + # Hard-fail here rather than inside the generators so the earlier repo- + # aggregation results are still printed before we exit non-zero. + print(f"\n{'='*60}") + print("Fetching GitHub releases...") + print(f"{'='*60}\n") + try: + gh_releases = list_repo_releases("gardenlinux", "gardenlinux") + except GitHubAPIError as exc: + print( + f"\nError: Could not fetch GitHub releases — {exc}\n" + "Hint: set GITHUB_TOKEN to avoid rate-limit issues: " + "export GITHUB_TOKEN=$(gh auth token)", + file=sys.stderr, + ) + return 1 + existing_gh_tags = {r["tag_name"].lstrip("v") for r in gh_releases} + print(f" Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)") + # Generate release documentation from GLRD print(f"\n{'='*60}") print("Generating release documentation...") print(f"{'='*60}\n") - generate_release_docs(docs_dir) + generate_release_docs(docs_dir, existing_gh_tags) - # Generate release notes from GitHub + # Generate release notes from pre-fetched GitHub releases print(f"\n{'='*60}") - print("Fetching release notes from GitHub...") + print("Generating release notes from GitHub...") print(f"{'='*60}\n") - generate_release_notes_docs(docs_dir) + generate_release_notes_docs(docs_dir, gh_releases) # Summary print(f"\n{'='*60}") diff --git a/src/aggregation/github_api.py b/src/aggregation/github_api.py new file mode 100644 index 0000000..47e499a --- /dev/null +++ b/src/aggregation/github_api.py @@ -0,0 +1,169 @@ +"""GitHub API HTTP client for docs-ng aggregation. + +This module is the single entry point for all GitHub HTTP calls in docs-ng. +Future callers should use :func:`get_json` or :func:`list_repo_releases` rather +than writing ad-hoc HTTP requests. + +Authentication +-------------- +Set the ``GITHUB_TOKEN`` environment variable to a personal access token (PAT) +or an OAuth token. No scopes are required for accessing public repositories. +When ``GITHUB_TOKEN`` is not set (or is empty after stripping whitespace), +requests are sent unauthenticated; GitHub applies a rate limit of 60 requests +per hour for unauthenticated access vs. 5 000 per hour for authenticated access. + +Recommended for local development:: + + export GITHUB_TOKEN=$(gh auth token) + +Failure handling +---------------- +Any network error, non-2xx HTTP status, rate-limit response (403 with +``X-RateLimit-Remaining: 0``), or JSON decode error raises +:class:`GitHubAPIError`. Callers should let this propagate so the make target +can hard-fail with a non-zero exit code. +""" + +import json +import os +import urllib.error +import urllib.request +from typing import Any + +GITHUB_API_BASE = "https://api.github.com" + + +class GitHubAPIError(Exception): + """Raised when any GitHub API request fails. + + The message includes the HTTP status code (when available) and the values + of the ``X-RateLimit-Remaining`` and ``X-RateLimit-Reset`` response headers + so that CI logs surface actionable information immediately. + """ + + +def _build_request(url: str) -> urllib.request.Request: + """Build a :class:`urllib.request.Request` with standard GitHub API headers. + + Adds ``Authorization: Bearer `` when ``GITHUB_TOKEN`` is set and + non-empty. + """ + headers = { + "Accept": "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + "User-Agent": "gardenlinux-docs-ng", + } + + token = os.environ.get("GITHUB_TOKEN", "").strip() + if token: + headers["Authorization"] = f"Bearer {token}" + + return urllib.request.Request(url, headers=headers) + + +def get_json(url: str) -> Any: + """Perform a GET request to *url* and return the parsed JSON body. + + Args: + url: Fully-qualified HTTPS URL of the GitHub API endpoint. + + Returns: + Parsed JSON value (usually a :class:`list` or :class:`dict`). + + Raises: + GitHubAPIError: On any network error, non-2xx HTTP status, or JSON + decode failure. The error message includes rate-limit headers when + present. + """ + req = _build_request(url) + + try: + with urllib.request.urlopen(req) as response: + body = response.read().decode("utf-8") + except urllib.error.HTTPError as exc: + # Extract rate-limit info from error headers when available. + rate_remaining = None + rate_reset = None + if exc.headers: + rate_remaining = exc.headers.get("X-RateLimit-Remaining") + rate_reset = exc.headers.get("X-RateLimit-Reset") + + msg = f"GitHub API request failed: HTTP {exc.code} for {url}" + if rate_remaining is not None: + msg += f" (X-RateLimit-Remaining: {rate_remaining}" + if rate_reset is not None: + msg += f", X-RateLimit-Reset: {rate_reset}" + msg += ")" + if rate_remaining == "0": + msg += ( + " — rate limit exhausted; set GITHUB_TOKEN to raise the limit " + "(export GITHUB_TOKEN=$(gh auth token))" + ) + raise GitHubAPIError(msg) from exc + except urllib.error.URLError as exc: + raise GitHubAPIError( + f"GitHub API request failed (network error) for {url}: {exc.reason}" + ) from exc + + try: + return json.loads(body) + except json.JSONDecodeError as exc: + raise GitHubAPIError( + f"GitHub API returned invalid JSON for {url}: {exc}" + ) from exc + + +def list_repo_releases( + owner: str, repo: str, per_page: int = 100 +) -> list[dict]: + """Fetch every release for *owner*/*repo* from the GitHub Releases API. + + Paginates automatically until GitHub returns an empty page. The full list + is returned only after all pages have been fetched successfully; a partial + list is never returned on error. + + Args: + owner: GitHub organisation or user name (e.g. ``"gardenlinux"``). + repo: Repository name (e.g. ``"gardenlinux"``). + per_page: Number of releases to request per page (max 100). + + Returns: + List of release objects as returned by the GitHub Releases API. + + Raises: + GitHubAPIError: On any fetch failure *or* when the first page is empty + (indicating the repository has no releases, which would break the + generated documentation). See module docstring for ``GITHUB_TOKEN`` + advice. + """ + all_releases: list[dict] = [] + page = 1 + + while True: + url = ( + f"{GITHUB_API_BASE}/repos/{owner}/{repo}/releases" + f"?per_page={per_page}&page={page}" + ) + page_data = get_json(url) + + if not isinstance(page_data, list): + raise GitHubAPIError( + f"GitHub API returned unexpected type for releases page {page}: " + f"{type(page_data).__name__}" + ) + + if len(page_data) == 0: + break + + all_releases.extend(page_data) + page += 1 + + if not all_releases: + raise GitHubAPIError( + f"GitHub API returned zero releases for {owner}/{repo}. " + "If this is unexpected, check that the repository exists and that " + "GITHUB_TOKEN (if set) has access to it. " + "Set GITHUB_TOKEN to authenticate: export GITHUB_TOKEN=$(gh auth token)" + ) + + return all_releases diff --git a/src/aggregation/release_notes.py b/src/aggregation/release_notes.py index 8af3eb2..2d2881b 100644 --- a/src/aggregation/release_notes.py +++ b/src/aggregation/release_notes.py @@ -1,8 +1,6 @@ -"""Fetch and format release notes from GitHub.""" +"""Format release notes from pre-fetched GitHub release data.""" -import json import re -import subprocess import sys from datetime import datetime from pathlib import Path @@ -10,12 +8,10 @@ from .glrd import get_active_minor_versions from .transformer import cleanup_github_markdown -GITHUB_API_URL = "https://api.github.com/repos/gardenlinux/gardenlinux/releases" GITHUB_RELEASES_URL = "https://github.com/gardenlinux/gardenlinux/releases/tag" GITHUB_COMMITS_URL = "https://github.com/gardenlinux/gardenlinux/commit" # Configuration -MAX_RELEASES = 200 # Include up to 200 recent releases ARCHIVED_DIR = "archived" @@ -55,39 +51,6 @@ def sort_by_version(releases: list) -> list: ) -def fetch_github_releases(per_page: int = 100) -> list: - """Fetch releases from GitHub API using curl with pagination.""" - all_releases = [] - page = 1 - max_pages = (MAX_RELEASES // per_page) + 2 # Fetch enough pages - - while page <= max_pages and len(all_releases) < MAX_RELEASES: - try: - result = subprocess.run( - [ - "curl", - "-s", - "-L", - f"{GITHUB_API_URL}?per_page={per_page}&page={page}", - ], - capture_output=True, - text=True, - check=False, - ) - if result.returncode == 0: - page_releases = json.loads(result.stdout) - if not page_releases: - break - all_releases.extend(page_releases) - else: - break - except (json.JSONDecodeError, Exception): - break - page += 1 - - return all_releases[:MAX_RELEASES] - - def format_release_date(date_str: str) -> str: """Format ISO date to readable format.""" if not date_str: @@ -129,8 +92,21 @@ def format_release_entry(release: dict) -> str: return f"{header}{body.strip()}\n" -def generate_release_notes_docs(docs_dir: Path) -> bool: - """Fetch GitHub release notes and generate release-notes as individual files.""" +def generate_release_notes_docs(docs_dir: Path, releases: list[dict]) -> bool: + """Generate release-notes documentation files from a pre-fetched release list. + + The caller (``aggregate.py``) is responsible for fetching the full release + list via :func:`aggregation.github_api.list_repo_releases` before calling + this function. No network requests are made here. + + Args: + docs_dir: Root of the VitePress ``docs/`` tree. + releases: Full list of GitHub release objects as returned by the GitHub + Releases API (already fetched and validated by the caller). + + Returns: + ``True`` on success, ``False`` if no releases were provided. + """ releases_dir = docs_dir / "reference" / "releases" / "release-notes" releases_dir.mkdir(parents=True, exist_ok=True) @@ -143,11 +119,8 @@ def generate_release_notes_docs(docs_dir: Path) -> bool: md_file.unlink() print(f" Removed: {md_file.relative_to(docs_dir)}") - print("Fetching release notes from GitHub...") - releases = fetch_github_releases() - if not releases: - print("Warning: No releases fetched from GitHub", file=sys.stderr) + print("Warning: No releases provided to generate_release_notes_docs", file=sys.stderr) return False # Query GLRD to determine release status @@ -169,9 +142,6 @@ def generate_release_notes_docs(docs_dir: Path) -> bool: # Sort by semantic version (highest first) filtered = sort_by_version(filtered) - # Limit - filtered = filtered[:MAX_RELEASES] - # Generate individual files for each release release_list = [] for idx, release in enumerate(filtered): @@ -186,9 +156,9 @@ def generate_release_notes_docs(docs_dir: Path) -> bool: r"^##\s+" + re.escape(name) + r"$", "# " + name, content, flags=re.MULTILINE ) - # Determine if this release is archived - # A release is active ONLY if it's explicitly in the active_versions dict - # All other releases are archived + # Determine if this release is archived. + # A release is active ONLY if it's explicitly in the active_versions set. + # All other releases are archived. tag_without_v = tag_name.lstrip("v") is_archived = tag_without_v not in active_versions diff --git a/src/aggregation/releases.py b/src/aggregation/releases.py index 9aa88a3..85bb8e8 100644 --- a/src/aggregation/releases.py +++ b/src/aggregation/releases.py @@ -1,4 +1,13 @@ -"""Generate release documentation from GLRD.""" +"""Generate release documentation from GLRD. + +The :func:`generate_release_docs` and :func:`generate_release_table` functions +accept a pre-built ``existing_gh_tags`` set that was fetched from the GitHub +Releases API by the caller (``aggregate.py``). GLRD rows that carry a +``minor`` version component are skipped when their normalized version string +(``{major}.{minor}[.{patch}]``) is absent from that set. Major-only rows are +always emitted. One warning line per skipped row is written to ``stderr`` so +that CI logs surface the omission. +""" import sys from pathlib import Path @@ -133,12 +142,26 @@ def generate_mermaid_gantt(releases_data: dict) -> str: return gantt -def generate_release_table(releases_data: dict, active_versions: set[str]) -> str: +def generate_release_table( + releases_data: dict, + active_versions: set[str], + existing_gh_tags: set[str], +) -> str: """Generate markdown table from GLRD JSON data. + GLRD rows that carry a ``minor`` version component are only included when + their normalized version string (``{major}.{minor}[.{patch}]``) appears in + *existing_gh_tags*. Rows without a matching GitHub release tag are skipped + and a warning is written to ``stderr``. Major-only rows are always emitted + regardless of *existing_gh_tags*. + Args: - releases_data: Release data from GLRD - active_versions: Set of active minor version strings + releases_data: Release data from GLRD. + active_versions: Set of active minor version strings (used for link + routing, not for inclusion filtering). + existing_gh_tags: Set of GitHub release tag names with leading ``v`` + stripped (e.g. ``{"1877.14", "2150.1.0"}``). Built from the full + list fetched by :func:`aggregation.github_api.list_repo_releases`. """ if not releases_data or "releases" not in releases_data: return "*No releases found*" @@ -151,6 +174,25 @@ def generate_release_table(releases_data: dict, active_versions: set[str]) -> st table += "|:--------|:-------|:---------------------|:---------------------|:-------------------|\n" for release in releases: + version_obj = release.get("version", {}) + has_minor = "minor" in version_obj + + if has_minor: + # Build the normalized version string for tag comparison. + if "patch" in version_obj: + version_str = ( + f"{version_obj['major']}.{version_obj['minor']}.{version_obj['patch']}" + ) + else: + version_str = f"{version_obj['major']}.{version_obj['minor']}" + + if version_str not in existing_gh_tags: + print( + f"Warning: skipping GLRD version {version_str} — no GitHub release found", + file=sys.stderr, + ) + continue + _, version_link = format_version(release, active_versions) commit_link = format_commit(release) @@ -199,8 +241,20 @@ def append_release_page( """ -def generate_release_docs(docs_dir: Path) -> bool: - """Fetch release data from GLRD and generate release documentation pages.""" +def generate_release_docs(docs_dir: Path, existing_gh_tags: set[str]) -> bool: + """Fetch release data from GLRD and generate release documentation pages. + + GLRD minor releases are only included in the generated tables when a + matching GitHub release tag exists in *existing_gh_tags* (see + :func:`generate_release_table`). + + Args: + docs_dir: Root of the VitePress ``docs/`` tree. + existing_gh_tags: Set of GitHub release tag names with leading ``v`` + stripped. Built from the full list fetched by the caller + (``aggregate.py``) via + :func:`aggregation.github_api.list_repo_releases`. + """ releases_dir = docs_dir / "reference" / "releases" releases_dir.mkdir(parents=True, exist_ok=True) @@ -220,7 +274,7 @@ def generate_release_docs(docs_dir: Path) -> bool: ) return False - active_table = generate_release_table(active_data, active_versions) + active_table = generate_release_table(active_data, active_versions, existing_gh_tags) active_gantt = generate_mermaid_gantt(active_data) active_timeline = get_timeline_section(active_gantt, "Release Timeline") @@ -252,7 +306,7 @@ def generate_release_docs(docs_dir: Path) -> bool: print(f" Updated: {release_path}") if archived_data is not None: - archived_table = generate_release_table(archived_data, active_versions) + archived_table = generate_release_table(archived_data, active_versions, existing_gh_tags) archived_gantt = generate_mermaid_gantt(archived_data) archived_timeline = get_timeline_section( archived_gantt, "Archived Releases Timeline" diff --git a/tests/unit/test_github_api.py b/tests/unit/test_github_api.py new file mode 100644 index 0000000..fccea7b --- /dev/null +++ b/tests/unit/test_github_api.py @@ -0,0 +1,208 @@ +"""Unit tests for aggregation.github_api module.""" + +import io +import json +import os +import urllib.error +import urllib.request +from http.client import HTTPMessage +from unittest.mock import MagicMock, patch + +import pytest + +from aggregation.github_api import GitHubAPIError, get_json, list_repo_releases + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _make_response(body: bytes, status: int = 200) -> MagicMock: + """Return a mock context-manager response that yields *body*.""" + resp = MagicMock() + resp.read.return_value = body + resp.__enter__ = lambda s: s + resp.__exit__ = MagicMock(return_value=False) + return resp + + +def _make_http_error(code: int, headers: dict | None = None) -> urllib.error.HTTPError: + """Build a :class:`urllib.error.HTTPError` with optional headers.""" + msg = HTTPMessage() + if headers: + for key, val in headers.items(): + msg[key] = val + return urllib.error.HTTPError( + url="https://api.github.com/test", + code=code, + msg=f"HTTP Error {code}", + hdrs=msg, + fp=io.BytesIO(b""), + ) + + +# --------------------------------------------------------------------------- +# get_json +# --------------------------------------------------------------------------- + +class TestGetJson: + """Tests for get_json.""" + + def test_returns_parsed_json(self): + """Successful response returns parsed JSON body.""" + payload = [{"tag_name": "1.0"}] + mock_resp = _make_response(json.dumps(payload).encode()) + + with patch("urllib.request.urlopen", return_value=mock_resp): + result = get_json("https://api.github.com/repos/x/y/releases") + + assert result == payload + + def test_no_auth_header_without_token(self, monkeypatch): + """No Authorization header is set when GITHUB_TOKEN is absent.""" + monkeypatch.delenv("GITHUB_TOKEN", raising=False) + + captured_req = {} + + def fake_urlopen(req): + captured_req["req"] = req + return _make_response(b"[]") + + with patch("urllib.request.urlopen", side_effect=fake_urlopen): + get_json("https://api.github.com/test") + + assert "Authorization" not in captured_req["req"].headers + + def test_no_auth_header_with_empty_token(self, monkeypatch): + """No Authorization header is added when GITHUB_TOKEN is empty / whitespace.""" + monkeypatch.setenv("GITHUB_TOKEN", " ") + + captured_req = {} + + def fake_urlopen(req): + captured_req["req"] = req + return _make_response(b"[]") + + with patch("urllib.request.urlopen", side_effect=fake_urlopen): + get_json("https://api.github.com/test") + + assert "Authorization" not in captured_req["req"].headers + + def test_auth_header_present_with_token(self, monkeypatch): + """Authorization header is set when GITHUB_TOKEN is non-empty.""" + monkeypatch.setenv("GITHUB_TOKEN", "ghp_testtoken") + + captured_req = {} + + def fake_urlopen(req): + captured_req["req"] = req + return _make_response(b"[]") + + with patch("urllib.request.urlopen", side_effect=fake_urlopen): + get_json("https://api.github.com/test") + + assert captured_req["req"].get_header("Authorization") == "Bearer ghp_testtoken" + + def test_raises_on_http_error(self): + """HTTPError is wrapped in GitHubAPIError.""" + with patch("urllib.request.urlopen", side_effect=_make_http_error(404)): + with pytest.raises(GitHubAPIError, match="HTTP 404"): + get_json("https://api.github.com/test") + + def test_raises_on_rate_limit_403(self): + """403 with zero remaining rate-limit mentions GITHUB_TOKEN in the error.""" + err = _make_http_error( + 403, + {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": "1700000000"}, + ) + with patch("urllib.request.urlopen", side_effect=err): + with pytest.raises(GitHubAPIError, match="GITHUB_TOKEN"): + get_json("https://api.github.com/test") + + def test_raises_on_url_error(self): + """URLError (network failure) is wrapped in GitHubAPIError.""" + net_err = urllib.error.URLError(reason="Name or service not known") + with patch("urllib.request.urlopen", side_effect=net_err): + with pytest.raises(GitHubAPIError, match="network error"): + get_json("https://api.github.com/test") + + def test_raises_on_invalid_json(self): + """Invalid JSON body raises GitHubAPIError.""" + with patch("urllib.request.urlopen", return_value=_make_response(b"not-json")): + with pytest.raises(GitHubAPIError, match="invalid JSON"): + get_json("https://api.github.com/test") + + +# --------------------------------------------------------------------------- +# list_repo_releases +# --------------------------------------------------------------------------- + +class TestListRepoReleases: + """Tests for list_repo_releases.""" + + def test_single_page(self): + """Single page of releases — returns the page contents.""" + page1 = [{"tag_name": "1.0"}, {"tag_name": "2.0"}] + + responses = [ + _make_response(json.dumps(page1).encode()), + _make_response(json.dumps([]).encode()), + ] + + with patch("urllib.request.urlopen", side_effect=responses): + result = list_repo_releases("owner", "repo") + + assert result == page1 + + def test_multiple_pages_concatenated(self): + """Multiple pages are concatenated into one list.""" + page1 = [{"tag_name": f"{i}.0"} for i in range(100)] + page2 = [{"tag_name": f"{i}.0"} for i in range(100, 150)] + + responses = [ + _make_response(json.dumps(page1).encode()), + _make_response(json.dumps(page2).encode()), + _make_response(json.dumps([]).encode()), + ] + + with patch("urllib.request.urlopen", side_effect=responses): + result = list_repo_releases("owner", "repo") + + assert len(result) == 150 + assert result == page1 + page2 + + def test_stops_on_empty_page(self): + """Pagination loop terminates on an empty page, not on IndexError.""" + page1 = [{"tag_name": "1.0"}] + + responses = [ + _make_response(json.dumps(page1).encode()), + _make_response(b"[]"), + ] + + with patch("urllib.request.urlopen", side_effect=responses): + result = list_repo_releases("owner", "repo") + + assert len(result) == 1 + + def test_raises_on_empty_first_page(self): + """Empty first page (repo has no releases) raises GitHubAPIError.""" + with patch("urllib.request.urlopen", return_value=_make_response(b"[]")): + with pytest.raises(GitHubAPIError, match="zero releases"): + list_repo_releases("owner", "repo") + + def test_raises_on_fetch_failure(self): + """GitHubAPIError from get_json propagates without swallowing.""" + with patch( + "urllib.request.urlopen", + side_effect=_make_http_error(500), + ): + with pytest.raises(GitHubAPIError): + list_repo_releases("owner", "repo") + + def test_raises_on_non_list_response(self): + """Non-list JSON (e.g. error dict) raises GitHubAPIError.""" + error_body = json.dumps({"message": "Not Found"}).encode() + with patch("urllib.request.urlopen", return_value=_make_response(error_body)): + with pytest.raises(GitHubAPIError, match="unexpected type"): + list_repo_releases("owner", "repo") diff --git a/tests/unit/test_release_filter.py b/tests/unit/test_release_filter.py new file mode 100644 index 0000000..ebad7bb --- /dev/null +++ b/tests/unit/test_release_filter.py @@ -0,0 +1,171 @@ +"""Unit tests for GLRD release-table filtering in aggregation.releases.""" + +import pytest + +from aggregation.releases import generate_release_table + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _make_release(major: int, minor: int | None = None, patch: int | None = None) -> dict: + """Build a minimal GLRD release dict.""" + version: dict = {"major": major} + if minor is not None: + version["minor"] = minor + if patch is not None: + version["patch"] = patch + return { + "version": version, + "git": {}, + "lifecycle": {}, + } + + +def _make_releases_data(*releases) -> dict: + return {"releases": list(releases)} + + +# --------------------------------------------------------------------------- +# Filtering: minor rows +# --------------------------------------------------------------------------- + +class TestGenerateReleaseTableFiltering: + """generate_release_table skips minor rows absent from existing_gh_tags.""" + + def test_minor_row_included_when_tag_present(self): + """A minor release present in existing_gh_tags appears in the table.""" + data = _make_releases_data(_make_release(1877, 14)) + table = generate_release_table(data, set(), existing_gh_tags={"1877.14"}) + assert "1877.14" in table + + def test_minor_row_skipped_when_tag_absent(self): + """A minor release absent from existing_gh_tags is dropped.""" + data = _make_releases_data(_make_release(1877, 14)) + table = generate_release_table(data, set(), existing_gh_tags=set()) + assert "1877.14" not in table + # Table should be effectively empty (header rows only) + data_rows = [ + line for line in table.splitlines() + if line.startswith("|") and not line.startswith("|:") and "Version" not in line + ] + assert data_rows == [] + + def test_major_row_always_included(self): + """A major-only row is emitted regardless of existing_gh_tags.""" + data = _make_releases_data(_make_release(1877)) + table = generate_release_table(data, set(), existing_gh_tags=set()) + assert "1877" in table + + def test_mixed_major_and_minor(self): + """Major rows pass through; unmatched minor rows are dropped.""" + data = _make_releases_data( + _make_release(2150), # major-only — always kept + _make_release(2150, 1, 0), # minor — tag present + _make_release(2150, 2, 0), # minor — tag absent + ) + table = generate_release_table( + data, + active_versions={"2150.1.0"}, + existing_gh_tags={"2150.1.0"}, + ) + assert "2150.1.0" in table + assert "2150.2.0" not in table + # Major row 2150 is present (as plain text, no dot-separated minor) + assert "2150" in table + + def test_v_prefix_stripped_from_gh_tag(self): + """Tags stored with a leading 'v' in existing_gh_tags are still matched.""" + # The caller strips 'v' before building the set, but simulate the raw + # comparison working by stripping manually (documents the contract). + raw_tags = {"v1877.14", "v2150.1.0"} + normalized = {t.lstrip("v") for t in raw_tags} + + data = _make_releases_data(_make_release(1877, 14)) + table = generate_release_table(data, set(), existing_gh_tags=normalized) + assert "1877.14" in table + + def test_patch_version_compared_correctly(self): + """Minor+patch version strings are matched against existing_gh_tags.""" + data = _make_releases_data(_make_release(2150, 1, 0)) + table = generate_release_table(data, set(), existing_gh_tags={"2150.1.0"}) + assert "2150.1.0" in table + + def test_patch_version_absent_skipped(self): + """Minor+patch version absent from existing_gh_tags is skipped.""" + data = _make_releases_data(_make_release(2150, 1, 0)) + table = generate_release_table(data, set(), existing_gh_tags={"2150.1"}) + # "2150.1.0" != "2150.1" — strict equality after normalization + assert "2150.1.0" not in table + + def test_warning_written_to_stderr_for_skipped(self, capsys): + """A warning line is written to stderr for each skipped minor release.""" + data = _make_releases_data(_make_release(1877, 99)) + generate_release_table(data, set(), existing_gh_tags=set()) + captured = capsys.readouterr() + assert "1877.99" in captured.err + assert "Warning" in captured.err + + def test_no_warning_for_included_minor(self, capsys): + """No warning is written when the minor release tag is present.""" + data = _make_releases_data(_make_release(1877, 14)) + generate_release_table(data, set(), existing_gh_tags={"1877.14"}) + captured = capsys.readouterr() + assert "1877.14" not in captured.err + + def test_empty_releases_data(self): + """Empty releases dict returns the 'no releases' sentinel.""" + table = generate_release_table({}, set(), existing_gh_tags=set()) + assert "*No releases found*" in table + + def test_empty_releases_list(self): + """Empty releases list returns the 'no releases' sentinel.""" + table = generate_release_table({"releases": []}, set(), existing_gh_tags=set()) + assert "*No releases found*" in table + + def test_glrd_release_exists_but_no_github_release(self): + """GLRD lists a minor release that has no corresponding GitHub release tag. + + This is the primary guard this filtering was designed for: a GLRD entry + whose tag was never pushed to GitHub (e.g. a planned but unpublished + release). The row must be dropped from the generated table and a + warning must appear on stderr. + """ + # GitHub has released 2150.0.0 and 2150.1.0 but NOT 2150.2.0. + # GLRD, however, already lists 2150.2.0 (e.g. an upcoming release). + existing_gh_tags = {"2150.0.0", "2150.1.0"} + data = _make_releases_data( + _make_release(2150, 0, 0), + _make_release(2150, 1, 0), + _make_release(2150, 2, 0), # in GLRD, but NO matching GitHub release + ) + table = generate_release_table( + data, + active_versions={"2150.1.0"}, + existing_gh_tags=existing_gh_tags, + ) + + # Releases that DO have GitHub tags are included. + assert "2150.0.0" in table + assert "2150.1.0" in table + # The GLRD-only entry is absent. + assert "2150.2.0" not in table + + def test_glrd_release_exists_but_no_github_release_emits_warning(self, capsys): + """A warning naming the missing version is written to stderr.""" + existing_gh_tags = {"2150.0.0"} + data = _make_releases_data( + _make_release(2150, 0, 0), + _make_release(2150, 1, 0), # GLRD only — no GitHub tag + ) + generate_release_table( + data, + active_versions=set(), + existing_gh_tags=existing_gh_tags, + ) + captured = capsys.readouterr() + assert "2150.1.0" in captured.err + assert "Warning" in captured.err + # The present release must NOT trigger a warning. + assert "2150.0.0" not in captured.err