feat(workflows): WorkflowResolver standalone (PR 1)

[markuswondrak]

Summary

This PR implements PR 1 of the workflow-overlays plan: a standalone WorkflowResolver that lets downstream projects extend installed workflows without editing the installed workflow.yml.

It follows the sequencing agreed with @mnriem in Discussion #3473: build the concrete workflow resolver first, keep it independent of PresetResolver, and defer any generic LayerStackResolver abstraction to a later PR.

What's included

• WorkflowResolver + StepListComposer (standalone, no shared abstraction)
• Pure-function merge engine supporting insert_after, insert_before, replace, remove
• Recursive anchor search inside then/else/steps/cases.*/default; fan-out templates excluded as discussed
• 2-tier layer sources: project overlays + base workflow
• workflow overlay add|set-priority|enable|disable|remove|list
• workflow resolve <id> for layer attribution
• Comprehensive tests under tests/workflows/

Architecture simplification

This PR enforces a clean separation of concerns:

• workflow add installs workflows only (no overlay copying)
• workflow overlay add installs overlays only (project-local)

Rationale: If upstream controls both the base workflow and shipped overlays, and both get overwritten on bundle update, there's no reason to ship overlays separately — just put those steps in the base workflow. Overlays only make sense when someone other than the base author adds them.

Review findings resolved

All three review findings from the initial PR review are now resolved:

• r3594064677: workflow add no longer copies overlays from all call sites (overlay-copying logic removed entirely)
• r3594064705: --priority override now applied before validation, not after (fixes timing issue where valid CLI priority couldn't override missing/invalid file priority)
• r3594064726: No stale installed overlays after reinstall (installed overlays tier removed entirely)

Test results

.venv/bin/python -m pytest tests/workflows -q
91 passed

.venv/bin/python -m pytest tests -q
3999 passed, 110 skipped

uv run ruff check src/specify_cli/workflows/overlays src/specify_cli/workflows/_commands.py src/specify_cli/workflows/engine.py tests/workflows
All checks passed!

tests/test_presets.py is untouched and remains green.

Scope notes

• PresetResolver is not modified.
• The spec/ planning folder is intentionally excluded from this PR.
• No new COMPONENT_KINDS entry; overlays are project-local only, matching the simplified 2-tier architecture.

AI assistance disclosure

This implementation was produced with AI assistance (Kimi / opencode-go/kimi-k2.7-code) and has been verified locally by running the full test suite and linting.

[Copilot]

Pull request overview

Implements the standalone WorkflowResolver agreed for PR 1 in Discussion #3473, adding upgrade-safe workflow overlays without modifying PresetResolver.

Changes:

• Adds layered workflow composition with recursive step edits and attribution.
• Adds overlay management and resolution CLI commands.
• Supports shipped overlays, documentation, and comprehensive tests.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/specify_cli/workflows/overlays/__init__.py	Defines the resolver facade.
src/specify_cli/workflows/overlays/composer.py	Composes and validates workflow layers.
src/specify_cli/workflows/overlays/layer_sources.py	Loads project, installed, and base layers.
src/specify_cli/workflows/overlays/merge.py	Implements step-list merge operations.
src/specify_cli/workflows/overlays/schema.py	Defines and validates overlay manifests.
src/specify_cli/workflows/overlays/_commands.py	Implements overlay CLI handlers.
src/specify_cli/workflows/engine.py	Resolves overlays when loading workflows.
src/specify_cli/workflows/_commands.py	Registers commands and installs shipped overlays.
docs/reference/workflows.md	Documents overlay behavior and commands.
tests/workflows/conftest.py	Adds shared workflow fixtures.
tests/workflows/test_overlay_commands.py	Tests overlay CLI operations.
tests/workflows/test_overlay_composer.py	Tests composition validation.
tests/workflows/test_overlay_merge.py	Tests merge behavior and attribution.
tests/workflows/test_overlay_schema.py	Tests manifest formats and validation.
tests/workflows/test_overlay_security.py	Tests path-handling protections.
tests/workflows/test_resolver_integration.py	Tests end-to-end resolution.

Review performed by GitHub Copilot.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mnriem]

@markuswondrak Looks like the right direction to me. Now we'll have to get it to the review cycles. So please address Copilot feedback and resolve conflicts

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 11 comments.

[mnriem]

Please address Copilot feedback and resolve conflicts

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

[mnriem]

Please address Copilot feedback

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/specify_cli/workflows/engine.py:749

• A missing YAML path now falls through to WorkflowResolver, which rejects the dot in missing.yml with ValueError. This changes load_workflow()'s documented missing-file contract from FileNotFoundError to “invalid workflow ID,” so callers can no longer distinguish an absent path from malformed workflow content. Treat any YAML-suffixed source as a path and raise FileNotFoundError before ID resolution when it does not exist.

        # Try as an installed workflow ID, resolving any overlays.
        resolver = WorkflowResolver(self.project_root)
        try:
            return resolver.resolve(str(source))

src/specify_cli/workflows/overlays/layer_sources.py:150

• Duplicate manifest IDs are all appended as active resolver layers, but _find_overlay_file() manages only the first sorted match. With aaa.yml and zzz.yml both declaring id: lint, disable, set-priority, or remove can report success after changing aaa.yml while zzz.yml remains active. Enforce unique manifest IDs consistently during loading/addition instead of allowing ambiguous layers.

            overlay, errors = validate_overlay_yaml(data)
            if overlay is None or errors:
                raise OverlayLoadError(path, errors)

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/specify_cli/workflows/overlays/_commands.py:200

• When no manifest ID matches, this blindly chooses <id>.yml. Since this API explicitly permits filenames to differ from manifest IDs, that filename may already hold a different overlay; the subsequent write_text silently destroys it. After the containment check, detect an existing candidate owned by another manifest and fail (or choose a collision-free filename) instead of overwriting it.

        target_path = _ensure_contained_path(
            target_dir / f"{overlay.id}.yml", _overlay_root(project_root)
        )

src/specify_cli/workflows/overlays/layer_sources.py:169

• Duplicate manifest IDs are accepted as separate layers and both are applied with the same project:<id> source, while _find_overlay_file returns only the first sorted match. Consequently disable/remove can report success while another overlay with the same ID remains active and cannot be addressed independently. Reject duplicate IDs as ambiguous during collection and make management lookup report the same error rather than codifying first-match behavior.

            layers.append(
                Layer(
                    content=overlay,
                    source=f"project:{overlay.id}",
                    tier=self.tier,
                    priority=overlay.priority,
                    path=path,
                )
            )

docs/reference/workflows.md:154

• This omits two names rejected by the implementation: _RESERVED_WORKFLOW_IDS also reserves runs and steps, so manifests documented as valid will fail validation. List all three reserved workflow IDs here.

| `extends` | yes | The workflow id this overlay applies to. Uses the same safe-id format as `id`; `overlays`, `runs`, and `steps` are reserved. |

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (2)

src/specify_cli/workflows/overlays/layer_sources.py:186

• This containment check trusts self.workflows_dir as the root, so if .specify or .specify/workflows itself is a symlink, both sides resolve outside the project and the check passes. BaseWorkflowSource.collect() can then read an external workflow when called directly, unlike ProjectOverlaySource, which validates every ancestor. Validate the base storage root and its ancestors against project_root before constructing the per-workflow path.

        _validate_workflow_id(workflow_id, self.workflows_dir)
        workflow_dir = self.workflows_dir / workflow_id
        _ensure_contained_dir(workflow_dir, self.workflows_dir)

src/specify_cli/workflows/overlays/_commands.py:119

• Returning the first manifest-ID match makes overlay identity ambiguous. The resolver still loads every file with that ID, so remove or set-priority can report success while another same-ID layer remains active, and workflow resolve attributes both as the same source. Detect duplicate manifest IDs and reject the ambiguous overlay set instead of silently selecting the first file.

        if data.get("id") == overlay_id:
            return path

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/specify_cli/workflows/engine.py:749

• load_workflow() documents YAML paths as valid sources and FileNotFoundError when such a file is missing, but a missing foo.yml now reaches resolver.resolve("foo.yml") and raises ValueError("Invalid workflow ID"). Handle a YAML-suffixed source that is not a file before invoking the ID resolver so the public API keeps distinguishing a missing path from an invalid installed ID; traversal-like IDs without a YAML suffix should still be rejected by the resolver.

        # Try as an installed workflow ID, resolving any overlays.
        resolver = WorkflowResolver(self.project_root)
        try:
            return resolver.resolve(str(source))

src/specify_cli/workflows/overlays/_commands.py:208

• Because mismatched filenames are explicitly supported, the fallback filename may already belong to a different manifest ID. For example, an existing ov1.yml with id: lint is not returned by _find_overlay_file(..., "ov1"); adding a new id: ov1 then silently overwrites and deletes the lint overlay. Refuse an occupied fallback path (or select an unused filename) before staging the write.

    # Reuse an existing .yaml file so we don't create a duplicate .yml layer.
    existing = _find_overlay_file(project_root, overlay.extends, overlay.id)
    if existing is not None:
        target_path = existing
    else:
        target_path = _ensure_contained_path(
            target_dir / f"{overlay.id}.yml", _overlay_root(project_root)
        )

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/specify_cli/workflows/overlays/layer_sources.py:105

• This validates symlinks but not other resolving aliases for the overlay root. On Windows, a directory junction can make overlays_root.resolve() point outside workflows_root while is_symlink() remains false; _ensure_contained_dir() then compares the child against that already-escaped root and accepts it, allowing resolver reads outside the project. Apply the same resolved containment check used by _resolve_workflows_root() before returning the overlay root.

    overlays_root = workflows_root / "overlays"
    if overlays_root.is_symlink():

src/specify_cli/workflows/overlays/_commands.py:54

• The CLI has the same containment gap as the resolver: a directory junction (or another resolving alias not reported by is_symlink()) can redirect .specify/workflows/overlays outside the project. Later checks only ensure targets remain under the already-escaped overlay root, so add/update/remove operations can write outside the project. Verify the resolved root remains under the resolved project root here.

    _reject_unsafe_workflow_storage(project_root)
    root = project_root / ".specify" / "workflows" / "overlays"
    _reject_unsafe_dir(root, ".specify/workflows/overlays")
    return root

[markuswondrak]

No more copilot findings.