Generalize BCS upload + register-build templates with repoName parameter#5241
Merged
LoopedBard3 merged 5 commits intoJun 12, 2026
Merged
Conversation
* Add repoName parameter (defaults to 'runtime' for back-compat) that
controls the {repoName} segment of the BCS blob path. Lets
non-runtime perf-build pipelines (starting with dotnet/aspnetcore)
upload to builds/{repoName}/buildArtifacts/{sha}/{configKey}/{file}.
* Add optional poolName/poolImage/poolOs parameters to
templates/upload-build-artifacts-job.yml. The historical runtime
callsite passes none, preserving the old behavior; pipelines
extending 1ES.Official (which requires every job to declare a pool)
pass them explicitly.
* Add 5 aspnetcore dispatch cases to upload-build-artifacts-jobs.yml
for aspnetcore_{x64,arm64}_linux and aspnetcore_{x64,arm64,x86}_windows.
dependencyJobName values match the jobName declared in the corresponding
dotnet/aspnetcore perf-build.yml; artifactName values match the artifact
`name:` published by those build jobs.
The dotnet/runtime perf-build.yml callsite needs no changes:
repoName defaults to 'runtime' and no pool* parameters are passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Generalizes the build-artifacts upload pipeline templates so multiple repositories/tenants (e.g., dotnet/aspnetcore) can upload to tenant-specific BCS paths and (optionally) declare explicit agent pools required by 1ES.Official-based pipelines.
Changes:
- Add
repoNameparameter to route uploads tobuilds/{repoName}/buildArtifacts/...(defaulting toruntimefor back-compat). - Add optional
poolName/poolImage/poolOsparameters and conditionally emit apool:block. - Add new dispatcher cases for aspnetcore build types (linux/windows, multiple architectures) that invoke the per-artifact upload job.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| eng/pipelines/upload-build-artifacts-jobs.yml | Adds repoName and optional pool parameters, forwards them to per-artifact jobs, and introduces aspnetcore dispatch cases. |
| eng/pipelines/templates/upload-build-artifacts-job.yml | Adds repoName and optional pool support; updates BCS upload path to use repoName. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Mirrors the repoName parameterization added to upload-build-artifacts-job.yml in the prior commit. Aspnetcore (and any other future tenant) needs its per-configKey buildInfo.json markers to land under builds/aspnetcore/... to match the layout written by the upload primitive — otherwise crank/UI tooling that reads buildInfo.json would have to special-case the path. Defaults to 'runtime' so the existing dotnet/runtime perf-build callsite is unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
changed the title
…s-job.yml The original 'Generalize upload-build-artifacts-job.yml for multi-tenant BCS uploads' commit (a8e0454) inadvertently pasted the entire upload steps block (checkout / download / each fileName / az storage blob upload) twice. Without this fix every per-artifact upload job would download the same pipeline artifact twice and run 'az storage blob upload' twice (--overwrite true so the second upload would no-op or stomp the first, both wasteful and confusing in build logs). Bisecting / cleanup tools that count expected upload steps per build would also report misleading numbers. Drops 15 duplicate lines. No behavioral change vs. the intended single-upload-per-artifact contract that runtime's pipeline relies on today. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When upload-build-artifacts-job.yml was first generalized in a8e0454, I added optional poolName/poolImage/poolOs params + a conditional `${{ if parameters.poolName }}: pool:` clause anticipating that the aspnetcore perf-build would extend 1ES.Official and therefore need each upload job to declare its own pool (1ES.Official requires it). That requirement is gone: aspnetcore's perf-build.yml no longer extends 1ES.Official — it matches dotnet/runtime's plain non-1ES shape, since these per-commit BCS uploads are private cache content (not signed or redistributed). Neither current caller passes pool params: * dotnet/runtime/eng/pipelines/performance/perf-build.yml -> passes runtimeRepoAlias, performanceRepoAlias, buildType only. * dotnet/aspnetcore/.azure/pipelines/perf-build.yml -> passes performanceRepoAlias, repoName, buildType only. With no caller setting poolName, the `${{ if parameters.poolName }}` clause always evaluates false, so the generated YAML is identical to what we'd produce without the clause. Removing the params and the clause is a pure refactor. Net change in this commit: * upload-build-artifacts-job.yml: -17 lines (param block + pool conditional) * upload-build-artifacts-jobs.yml: -79 lines (param block + 3 passthrough lines x 23 dispatch cases) If a future tenant ever needs 1ES.Official, re-adding these is a few lines of YAML and a one-line caller update — cheap to bring back. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
changed the title
Open
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make the build registration and upload steps generalized by repo for the BuildCacheService so that we can manage the steps in the dotnet/performance repo instead of requiring each setup we end up with having to duplicate the work. This keeps the defaults to runtime so that the runtime ymls continue working.
AI Summary:
Companion to a forthcoming dotnet/aspnetcore PR (LoopedBard3/aspnetcore#2 in fork-staging) that wires aspnetcore per-commit perf builds into BCS using the templates this PR generalizes.
Why
Today the BCS-upload templates in
eng/pipelines/are hard-tied to dotnet/runtime: bothtemplates/upload-build-artifacts-job.ymlandtemplates/register-build-job.ymlwrite to literalbuilds/runtime/...blob paths. To let dotnet/aspnetcore (and any future tenant) consume the same primitives — and produce a BCS layout that's uniform with runtime's, so a single dashboard / cleanup tool / future indexer code path can serve both — we just need to parameterize the path prefix.What changes
eng/pipelines/templates/upload-build-artifacts-job.yml(per-artifact upload primitive):repoName(default'runtime', back-compat for existing runtime callers).builds/runtime/withbuilds/${{ parameters.repoName }}/.eng/pipelines/templates/register-build-job.yml(per-configbuildInfo.jsonmarker):repoName(default'runtime').builds/runtime/withbuilds/${{ parameters.repoName }}/.eng/pipelines/upload-build-artifacts-jobs.ymlandeng/pipelines/register-build-jobs.yml(dispatchers):repoNameinto each per-config invocation.aspnetcore_{x64,arm64}_linux+aspnetcore_{x64,arm64,x86}_windows), each depending on the correspondingWindows_build/Linux_x64_build/Linux_arm64_buildjob that the aspnetcore perf-build defines viadefault-build.yml@self. (RegisterBuild's dispatcher is buildType-agnostic by construction so it needs no new cases.)What we deliberately did NOT add
poolName/poolImage/poolOsparams. An earlier draft of this PR added optional pool params to the upload leaf so a 1ES.Official-extending tenant could declare a per-job pool (1ES requires every job to have one). aspnetcore's caller in Add Status Tracking Jobs #2 dropped 1ES.Official to match runtime's plain non-1ES shape (private cache content, not signed/redistributed), so no current or planned tenant needs them. Removed to keep the surface area minimal; if a future 1ES tenant ever lands, re-adding the params is ~10 lines of YAML.runtimeRepoAlias/mauiFrameworkcleanup. Both are pre-existing dead params on the upload dispatcher. Cleanup is split into a separate small PR (Clean up unused parameter and stray escape in upload template #5242) to keep this one focused on generalization.Validation
python -c "import yaml; yaml.safe_load(...)"for all four files.dotnet/runtimeeng/pipelines/performance/perf-build.yml) is unchanged — it calls both dispatchers withoutrepoName, the default'runtime'kicks in, output is identical to today.repoName: aspnetcoreto both dispatchers.Why a single shared template (not a fork)
Earlier plan iterations considered duplicating the upload logic in dotnet/aspnetcore. After reading runtime's actual
perf-build.ymlstructure, it's clear runtime ALREADY consumes a thin-wrapper-from-dotnet-performance architecture (register-build-jobs.yml@performance,runtime-perf-build-jobs.yml@performance,upload-build-artifacts-jobs.yml@performance). Generalizing the existing primitives instead of forking keeps a single source of truth for the BCS contract, and means future BCS-shape changes land once for all tenants.Sequencing
This PR must merge upstream before the aspnetcore perf-build pipeline can run end-to-end — the aspnetcore-side YAML resolves the templates via
@performance(=internal/dotnet-performanceAzDO mirror), which only updates after dotnet/performance's main branch advances.