Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
root = true

[*.py]
profile = black
indent_style = space
indent_size = 4
src_paths = src,tests

3 changes: 3 additions & 0 deletions .isort.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[settings]
profile=black
src_paths=src,tests
18 changes: 16 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help run build preview aggregate aggregate-repo aggregate-update-repo aggregate-repo-single aggregate-update-repo-single test-aggregate-local clean clean-projects clean-aggregated-git test test-unit test-integration check spelling linkcheck woke
.PHONY: help run build preview aggregate aggregate-repo aggregate-update-repo aggregate-repo-single aggregate-update-repo-single test-aggregate-local clean clean-projects clean-aggregated-git test test-unit test-integration check spelling linkcheck woke glossary glossary-check format

help:
@echo "Garden Linux Documentation Hub - Available targets:"
Expand All @@ -18,6 +18,8 @@ help:
@echo " spelling - Check spelling with codespell"
@echo " linkcheck - Check links with lychee"
@echo " woke - Check inclusive language with woke"
@echo " glossary - Process glossary links in documentation"
@echo " glossary-check - Validate glossary structure"
@echo ""
@echo " Documentation Aggregation:"
@echo " aggregate-local - Aggregate from local repos (file:// URLs in repos-config.local.json)"
Expand All @@ -42,12 +44,16 @@ install:
run:
pnpm run docs:dev

build: install clean aggregate
build: install clean aggregate glossary
pnpm run docs:build

preview:
pnpm run docs:preview

format:
black src/ tests/
isort src/ tests/

# Testing
test: test-unit test-integration
@echo "All tests passed!"
Expand Down Expand Up @@ -76,6 +82,14 @@ woke:
@echo "Running inclusive language checks..."
@pnpm run docs:woke

glossary:
@echo "Processing glossary links..."
@python3 src/aggregation/auto_glossary.py docs/

glossary-check:
@echo "Validating glossary structure..."
@python3 src/aggregation/auto_glossary.py docs/ --check

# Documentation Aggregation
aggregate-local:
@echo "Aggregating from local repositories (relative paths)..."
Expand Down
211 changes: 211 additions & 0 deletions docs/contributing/documentation/auto-glossary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
---
title: "Documentation Auto Glossary"
description: "Automatically link common idioms to their glossary entry"
order: 13
related_topics:
- /contributing/documentation/documentation_workflow.md
- /contributing/documentation/writing_good_docs.md
- /contributing/documentation/adding-repos.md
- /contributing/documentation/working-locally.md
- /contributing/documentation/ci-architecture.md
- /contributing/documentation/ci-workflows-reference.md
- /contributing/documentation/configuration.md
- /contributing/documentation/technical.md
- /contributing/documentation/testing.md
- /contributing/documentation/vitepress-features.md
---

# Auto Glossary

The documentation system includes an automatic glossary linker that converts marked terms into links pointing to the glossary page.

## Quick Start

Mark terms in your documentation using the marker format:

```markdown
Deploy {glossary:Gardenlinux} on {glossary:AWS} using {glossary:KVM}.
```

After aggregation, this becomes:

```markdown
Deploy [Gardenlinux](/reference/glossary#gardenlinux) on [AWS](/reference/glossary#aws) using [KVM](/reference/glossary#kvm).
```

## Function

During aggregation (`make aggregate`), the system:

1. Parses `docs/reference/glossary.md` to extract all level-3 headers as terms
2. Extracts aliases from terms with parenthesized expansions (e.g., `ADR (Architecture Decision Record)`)
3. Scans all markdown files for glossary markers
4. Replaces markers with markdown links to glossary anchors
5. Preserves code blocks, inline code, and existing links

## Marking Terms

Use the format where the term name matches a glossary entry:

```markdown
{glossary:AWS}
{glossary:Garden Linux}
{glossary:ADR (Architecture Decision Record)}
```

Term matching is case-insensitive but preserves your original formatting:

- Input: `{glossary:aws}` produces `[aws](/reference/glossary#aws)`
- Input: `{glossary:AWS}` produces `[AWS](/reference/glossary#aws)`

::: details Developer Information
The auto glossary system is designed to accept a new custom `entry_format` through which
the shortcode `{glossary:*}` can be replaced with a new custom pattern.

Check the source code for more information.
:::
Comment thread
ByteOtter marked this conversation as resolved.

## Protected Regions

The system leaves certain content unchanged:

- Fenced code blocks (triple backticks)
- Inline code (single backticks)
- Existing markdown links

Glossary markers inside these regions are not processed.

## Adding Glossary Terms

To add a new term:

1. Edit `docs/reference/glossary.md`
2. Add a level-3 header:

```markdown
### New Term Name

Definition of the term.
```

The anchor is generated automatically (lowercase, hyphens replace spaces). Reference it with the marker format in your documentation.

## Alias Support

Terms with parenthesized expansions create aliases automatically.

Given this glossary entry:

```markdown
### ADR (Architecture Decision Record)
```

The system creates:
- Main term: `ADR (Architecture Decision Record)`
- Alias: `Architecture Decision Record`
- Alias: `ADR`

All three can be used in markers and link to the same glossary entry.

## Auto-Linking



Auto-linking creates links for glossary terms without requiring explicit markers. This feature is disabled by default.

With auto-linking enabled, input text:

```
Deploy Gardenlinux on AWS using KVM virtualization.
```

Becomes:

```markdown
Deploy [Garden Linux](/reference/glossary#garden-linux) on [AWS](/reference/glossary#aws) using [KVM](/reference/glossary#kvm) virtualization.
```

Auto-linking rules:
- Links only the first occurrence of each term
- Matches longer terms first (`Garden Linux` before `Linux`)
- Case-insensitive matching
- Respects word boundaries
- Preserves code blocks and inline code
- Does not modify existing links

::: warning
This feature is disabled by default as it is highly experimental and intended for research work only.

**Contributors must not rely on this feature when contributing documentation.**
:::
Comment on lines +110 to +140

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove this and also the feature until it is ready? It adds a lot of new code and in it's current state it is not very helpful.

I like the feature, though and would be happy if we get this included if it works properly.


## Makefile Targets

Process glossary links manually:

```bash
make glossary

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work for me currently. No files are touched.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed a bunch of bugs now regardingfile handling and how we deal with markdown formatting and protected regions. Please try again and see if it works. :)

```

Validate glossary structure:

```bash
make glossary-check
```

## Warnings

**Term not found:**

```
[Warning][auto-glossary] Term 'xyz' not found in glossary (referenced in file.md)
```

The marker remains unchanged. Add the term to the glossary, fix the spelling, or remove the marker.

**Anchor collision:**

```
[Warning][auto-glossary] Anchor collision detected:
'Term 1' and 'Term 2' both generate anchor 'term-1-2'
```

Rename one of the terms in the glossary.

## Anchor Generation

Terms are converted to VitePress-compatible anchors:

- Convert to lowercase
- Replace spaces and special characters with hyphens
- Collapse consecutive hyphens
- Strip leading and trailing hyphens
- Normalize Unicode to ASCII

Examples:
- `AWS` becomes `aws`
- `Garden Linux` becomes `garden-linux`
- `ADR (Architecture Decision Record)` becomes `adr-architecture-decision-record`

## Technical Reference

Module location: `src/aggregation/auto_glossary.py`

Main components:
- `AutoGlossary` class for glossary processing
- `process_glossary_links()` function for batch processing
- `GLOSSARY_ENTRY_FORMAT` constant defining the default marker format

Integration points:
- Runs as part of `src/aggregate.py` after release notes generation
- Makefile provides `glossary` and `glossary-check` targets

## Testing

Run the test suite:

```bash
python3 -m pytest tests/unit/test_auto_glossary*.py tests/unit/test_generate_anchor.py -v
```

Test coverage includes format validation, anchor generation, alias extraction, auto-linking, and integration tests.
2 changes: 2 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
codespell==2.4.2
black
isort
pytest
pyyaml
# glrd @ git+https://github.com/gardenlinux/glrd.git@v4.2.0
Expand Down
14 changes: 7 additions & 7 deletions src/aggregate.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@
import tempfile
from pathlib import Path

from aggregation import (DocsFetcher, copy_targeted_docs, load_config,
save_config)
from aggregation.structure import verify_internal_links
from aggregation import DocsFetcher, copy_targeted_docs, load_config, save_config
from aggregation.auto_glossary import process_glossary_links
from aggregation.flavor_matrix import generate_flavor_matrix_docs
from aggregation.github_api import GitHubAPIError, list_repo_releases
from aggregation.release_notes import generate_release_notes_docs
from aggregation.releases import generate_release_docs
from aggregation.structure import verify_internal_links


def transform_repo_docs(
Expand Down Expand Up @@ -227,9 +227,7 @@ def main() -> int:
repo_names = {repo.name for repo in repos}
if args.repo not in repo_names:
if args.single:
parser.error(
f"Repository '{args.repo}' not found in config"
)
parser.error(f"Repository '{args.repo}' not found in config")
else:
print(f"WARNING: Repository '{args.repo}' not found in config")

Expand Down Expand Up @@ -308,7 +306,9 @@ def main() -> int:
)
return 1
existing_gh_tags = {r["tag_name"].lstrip("v") for r in gh_releases}
print(f" Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)")
print(
f" Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)"
)

# Generate release documentation from GLRD
print(f"\n{'='*60}")
Expand Down
15 changes: 11 additions & 4 deletions src/aggregation/__init__.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
"""Aggregation package for docs-ng documentation aggregation."""

# Re-export commonly used functions for backward compatibility with tests
from .auto_glossary import AutoGlossary, process_glossary_links
from .config import load_config, save_config
from .fetcher import DocsFetcher
from .flavor_matrix import generate_flavor_matrix_docs
from .models import AggregateResult, RepoConfig
from .releases import generate_release_docs
from .sphinx_builder import build_sphinx_markdown
from .structure import (copy_targeted_docs, process_all_markdown,
verify_internal_links)
from .transformer import (ensure_frontmatter, parse_frontmatter,
quote_yaml_value, rewrite_links)
from .structure import copy_targeted_docs, process_all_markdown, verify_internal_links
from .transformer import (
ensure_frontmatter,
parse_frontmatter,
quote_yaml_value,
rewrite_links,
)

__all__ = [
# Models
Expand All @@ -36,4 +40,7 @@
"generate_flavor_matrix_docs",
# Sphinx builder
"build_sphinx_markdown",
# Auto Glossary
"AutoGlossary",
"process_glossary_links",
]
Loading
Loading