Skip to content

quality/coverage: enable Rust coverage in the LLVM pipeline - #772

Open
dcalavrezo-qorix wants to merge 2 commits into
mainfrom
dcalavrezo_llvm_cov_rust
Open

quality/coverage: enable Rust coverage in the LLVM pipeline#772
dcalavrezo-qorix wants to merge 2 commits into
mainfrom
dcalavrezo_llvm_cov_rust

Conversation

@dcalavrezo-qorix

Copy link
Copy Markdown
Until now the unified llvm-cov pipeline reported C++ only: the Ferrocene
toolchain registered for coverage runs did not declare llvm_cov, so
rules_rust never instrumented Rust crates, no .rs file was in the scope
allowlist, and justify.py did not scan Rust sources.

Toolchain:
- bump score_toolchains_rust to 0.9.2: its ferrocene extension now wires
  llvm-cov/llvm-profdata from the coverage-tools tarball into the
  generated rust_toolchain, which activates -Cinstrument-coverage under
  'bazel coverage'
- bump ferrocene_toolchain_builder coverage-tools to 1.3.0 (ships
  llvm-cov/llvm-profdata/llvm-cxxfilt built from the same LLVM as rustc)

coverage.bazelrc:
- -Cllvm-args=-runtime-counter-relocation: continuous-mode profiling
  (LLVM_PROFILE_CONTINUOUS_MODE) otherwise writes no .profraw for Rust
- -Clink-dead-code: keep unreferenced functions visible as uncovered
- -Zcoverage-options=branch: branch coverage instrumentation (unstable
  flag; works because Ferrocene is a nightly-based rolling build)

Pipeline:
- coverage_scope.bzl: CrateInfo branch collects rust_binary sources and
  executables; rust_library was already handled via CcInfo; remove
  leftover debug prints
- merger.py: resolve RUST_LLVM_PROFDATA (Bazel only sets LLVM_PROFDATA
  via the cc toolchain); pick up the Rust test binary from the coverage
  manifest (ELF detection), skipping external/ toolchain binaries
- reporter.py: normalize /proc/self/cwd/ covmap paths so the allowlist
  subtraction works with mixed C++ (absolute) and Rust (relative) paths;
  expand Rust rlib baseline archives into their .o members (llvm-cov
  rejects rlibs due to the leading lib.rmeta member); resolve
  llvm-cxxfilt via runfiles (demangles Rust v0/legacy and C++ symbols)
- justify.py: scan .rs files for COV_JUSTIFIED markers; skip bazel-*
  convenience symlinks
- quality/coverage/BUILD: add the Rust production crates to
  coverage_scope (mocks/test_support excluded); crate visibilities
  widened by //quality/coverage:__pkg__ only

Known limitations:
- com-api-concept-test and com-api-runtime-lola-tests are tagged
  'manual' and are skipped by wildcard coverage runs; their crates
  report as 0% baseline unless the tests are invoked explicitly
- lib.rs/error.rs/reloc.rs-style files without executable code carry no
  coverage regions and do not appear in reports (same as C++ headers)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the unified llvm-cov coverage pipeline to include Rust crates (in addition to C++), by updating the Rust toolchain configuration, broadening the coverage scope to Rust sources/binaries, and enhancing the merger/reporter/justification tooling so Rust artifacts and paths are handled correctly.

Changes:

  • Enable Rust coverage instrumentation under bazel coverage via Ferrocene/toolchain updates and Rust-specific coverage.bazelrc flags.
  • Extend coverage scoping and collection to Rust binaries/crates (allowlist + baseline objects), and update merger/reporter to handle Rust-specific profdata/object discovery and path normalization.
  • Update justification scanning to include .rs sources and avoid descending into Bazel output symlinks.

Reviewed changes

Copilot reviewed 5 out of 14 changed files in this pull request and generated no comments.

Show a summary per file
File Description
score/mw/com/impl/rust/com-api/com-api-runtime-lola/BUILD Widens visibility to allow //quality/coverage to include this crate in coverage scoping.
score/mw/com/impl/rust/com-api/com-api-ffi-lola/BUILD Widens visibility for coverage scoping access.
score/mw/com/impl/rust/com-api/com-api-concept/BUILD Widens visibility for coverage scoping access.
score/mw/com/impl/plumbing/rust/BUILD Widens visibility for coverage scoping access to plumbing Rust crates.
quality/coverage/llvm_cov/reporter.py Adds Rust path normalization, rlib baseline expansion, and runfiles-based llvm-cxxfilt resolution.
quality/coverage/llvm_cov/merger.py Resolves llvm-profdata for Rust runs and detects Rust test executables from the manifest.
quality/coverage/llvm_cov/BUILD Adds llvm-cxxfilt as a data dependency for demangling support.
quality/coverage/justify.py Scans .rs files for justification markers and skips Bazel convenience symlink trees.
quality/coverage/coverage.bazelrc Adds Rust-specific rustc flags needed for usable coverage output (incl. continuous-mode profiling).
quality/coverage/coverage_scope.bzl Extends scope collection to handle rust_binary via CrateInfo and removes debug prints.
quality/coverage/coverage_justifications.yaml Adds explicit platforms fields to select justifications.
quality/coverage/BUILD Explicitly adds Rust production crates to the overall coverage scope.
MODULE.bazel.lock Updates module lock entries for the new Rust toolchain + Ferrocene coverage tools.
MODULE.bazel Bumps score_toolchains_rust and updates Ferrocene coverage-tools URL/SHA to a newer release.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@castler castler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great contribution. Thanks! Some comments from my side.

Comment thread quality/coverage/llvm_cov/merger.py Outdated
Comment thread quality/coverage/BUILD Outdated
Comment thread quality/coverage/coverage.bazelrc Outdated
Comment thread score/mw/com/impl/plumbing/rust/BUILD Outdated
Comment thread quality/coverage/llvm_cov/reporter.py Outdated
Copilot AI review requested due to automatic review settings July 27, 2026 06:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 10 changed files in this pull request and generated no new comments.

@dcalavrezo-qorix
dcalavrezo-qorix requested a review from castler July 27, 2026 07:11
@castler

castler commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Thanks for addressing that. Do you maybe want to fixup your changes into the first commit and resolve the merge conflicts? Then we could try to merge that?

@dcalavrezo-qorix

Copy link
Copy Markdown
Author

Thanks for addressing that. Do you maybe want to fixup your changes into the first commit and resolve the merge conflicts? Then we could try to merge that?

@castler could you please re-review? changed the code according to your comments

@LittleHuba

Copy link
Copy Markdown
Contributor

@dcalavrezo-qorix
Please first fix existing merge conflicts.
Also squash your third commit into the first commit.
Then it makes sense to do another review from our side.

Copilot AI review requested due to automatic review settings July 28, 2026 13:03
@dcalavrezo-qorix
dcalavrezo-qorix force-pushed the dcalavrezo_llvm_cov_rust branch from 678c644 to 79e4e5c Compare July 28, 2026 13:03
@dcalavrezo-qorix

Copy link
Copy Markdown
Author

@dcalavrezo-qorix Please first fix existing merge conflicts. Also squash your third commit into the first commit. Then it makes sense to do another review from our side.

Rebased onto main and squashed the review-fix commit into the first commit as requested. Notes:

  • The scope now uses the new public //score/mw/com/rust:score_com from the Rust API restructure — which is exactly the "public Rust targets" this thread asked for; the aspect reaches all production crates transitively, and the visibility changes are gone from history.
  • While validating on top of current main I found two pre-existing coverage breakages there (reproducible on pristine origin/main): (1) Rust tests fail to link under bazel coverage (atexit undefined from libclang_rt.profile.a) — a side effect of the 0.9.2 toolchain bump half-enabling Rust coverage; this PR fixes it by completing the wiring. (2) _FORTIFY_SOURCE macro redefined + -Werror in exec-config compiles ([for tool]) breaks the coverage report generator's build — this appears to come from the compiler-warnings rework and is independent of this PR; happy to file it as an issue. Until (2) is fixed, the combined coverage report can't be generated on main or any branch.
  • The nightly quality failure from last night is a third, unrelated infra issue (403 fetching download_utils from gitlab.arm.com).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 10 changed files in this pull request and generated no new comments.

Until now the unified llvm-cov pipeline reported C++ only: the Ferrocene
toolchain registered for coverage runs did not declare llvm_cov, so
rules_rust never instrumented Rust crates, no .rs file was in the scope
allowlist, and justify.py did not scan Rust sources.

Toolchain (score_toolchains_rust is already at 0.9.2 on main, whose
ferrocene extension wires llvm-cov/llvm-profdata into the generated
rust_toolchain):
- bump ferrocene_toolchain_builder coverage-tools to 1.3.0, which ships
  llvm-cov/llvm-profdata/llvm-cxxfilt built from the same LLVM as rustc

coverage.bazelrc (LLVM pipeline only; reset to empty for the gcov-based
QNX pipeline via the list-typed extra_rustc_flags setting):
- -Cllvm-args=-runtime-counter-relocation: continuous-mode profiling
  (LLVM_PROFILE_CONTINUOUS_MODE) otherwise writes no .profraw for Rust
- -Clink-dead-code: keep unreferenced functions visible as uncovered
- -Zcoverage-options=branch: branch coverage instrumentation (unstable
  flag; works because Ferrocene is a nightly-based rolling build)

Pipeline:
- coverage_scope.bzl: CrateInfo branch collects rust_binary sources and
  executables; rust_library was already handled via CcInfo; remove
  leftover debug prints
- merger.py: resolve RUST_LLVM_PROFDATA (Bazel only sets LLVM_PROFDATA
  via the cc toolchain), terminating with a detailed error when no
  llvm-profdata is found; pick up the Rust test binary from the coverage
  manifest (ELF detection), skipping external/ toolchain binaries
- reporter.py: normalize /proc/self/cwd/ covmap paths so the allowlist
  subtraction works with mixed C++ (absolute) and Rust (relative) paths;
  expand Rust rlib baseline archives into their .o members (llvm-cov
  rejects rlibs due to the leading lib.rmeta member); demangle via
  llvm-cxxfilt from runfiles (Rust v0/legacy and C++ symbols),
  terminating with an error when the declared data dependency is missing
- justify.py: scan .rs files for COV_JUSTIFIED markers; skip bazel-*
  convenience symlinks
- quality/coverage/BUILD: add the public Rust API crate
  (//score/mw/com/rust:score_com) to coverage_scope; the scope aspect
  reaches all Rust production crates through its transitive deps

Known limitations:
- 'manual'-tagged Rust tests are skipped by wildcard coverage runs;
  their crates report as 0% baseline unless invoked explicitly
- Files without executable code (lib.rs-style re-export modules) carry
  no coverage regions and do not appear in reports (same as C++ headers)
justify.py validates 'platforms' as mandatory (as documented in
llvm_cov/README.md), but the three gateway-* entries were added without
it. Because generate_coverage_html.sh tolerates justify failures, the
justification and effective-coverage processing has been silently
skipped since then. With the field added, the 8 justified lines resolve
again.
@dcalavrezo-qorix
dcalavrezo-qorix force-pushed the dcalavrezo_llvm_cov_rust branch from 79e4e5c to 951d332 Compare July 28, 2026 13:16
@LittleHuba

Copy link
Copy Markdown
Contributor

Thanks for the update and the accompanying notes.

(2) _FORTIFY_SOURCE macro redefined + -Werror in exec-config compiles ([for tool]) breaks the coverage report generator's build — this appears to come from the compiler-warnings rework and is independent of this PR; happy to file it as an issue. Until (2) is fixed, the combined coverage report can't be generated on main or any branch.

If you already know what is necessary to fix this, feel free to add a commit to this PR. If not, please create a ticket and we will try to pick it up as soon as we can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

4 participants