Regression matrix: build unmodified repos against local FSharp.Core

.github/agents/compiler-perf-investigator.md

@@ -40,7 +40,7 @@ These are **general investigation instructions** for this agent, a template for
 ### 1. Preparation
 - **Setup:** Clone/generate repo/snippet/etc.
 - **Clear old config:** Remove `global.json` unless needed.
-- **Prepare local compiler:** Use `PrepareRepoForRegressionTesting.fsx` and absolute env paths.
+- **Prepare local compiler:** Build via `dotnet fsi <fsharp-repo>/eng/scripts/BuildWithLocalFSharp.fsx --build-script '<cmd>'` (sets the local-compiler + FSharp.Core shim env).
 
 ### 2. Experiment Matrix
 

UseLocalCompiler.Directory.Build.props

@@ -7,39 +7,51 @@
 
   <PropertyGroup>
     <LoadLocalFSharpBuild Condition="'$(LoadLocalFSharpBuild)' == ''">False</LoadLocalFSharpBuild>
-
     <LocalFSharpCompilerConfiguration Condition="'$(LocalFSharpCompilerConfiguration)' == ''">Release</LocalFSharpCompilerConfiguration>
+    <LocalFSharpCompilerPath Condition="'$(LocalFSharpCompilerPath)' == ''">$(MSBuildThisFileDirectory)</LocalFSharpCompilerPath>
+  </PropertyGroup>
 
-    <LocalFSharpCompilerPath Condition=" '$(LocalFSharpCompilerPath)' == '' ">$(MSBuildThisFileDirectory)</LocalFSharpCompilerPath>
-
-    <DisableImplicitFSharpCoreReference>true</DisableImplicitFSharpCoreReference>
+  <!-- Gate 1: redirect the SDK to the locally built F# compiler, build tasks and targets (via FSharpTargetsShim).
+       See: https://github.com/dotnet/sdk/blob/main/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FSharpTargetsShim.targets -->
+  <PropertyGroup Condition="'$(LoadLocalFSharpBuild)' == 'True'">
     <DisableAutoSetFscCompilerPath>true</DisableAutoSetFscCompilerPath>
     <FscToolPath Condition="'$(FscToolPath)' == ''">$([System.IO.Path]::GetDirectoryName($(DOTNET_HOST_PATH)))</FscToolPath>
     <FscToolExe Condition="'$(FscToolExe)' == ''">$([System.IO.Path]::GetFileName($(DOTNET_HOST_PATH)))</FscToolExe>
-
-    <DotnetFscCompilerPath>$(LocalFSharpCompilerPath)/artifacts/bin/fsc/$(LocalFSharpCompilerConfiguration)/$(FSharpNetCoreProductTargetFramework)/fsc.dll</DotnetFscCompilerPath>
-    <Fsc_DotNET_DotnetFscCompilerPath>$(LocalFSharpCompilerPath)/artifacts/bin/fsc/$(LocalFSharpCompilerConfiguration)/$(FSharpNetCoreProductTargetFramework)/fsc.dll</Fsc_DotNET_DotnetFscCompilerPath>
-
     <FSharpPreferNetFrameworkTools>False</FSharpPreferNetFrameworkTools>
     <FSharpPrefer64BitTools>True</FSharpPrefer64BitTools>
-  </PropertyGroup>
 
-  <!-- 
-    Use FSharpTargetsShim to redirect the SDK to use the locally built F# targets.
-    This replaces all the individual UsingTask overrides since Microsoft.FSharp.NetSdk.targets
-    imports Microsoft.FSharp.Targets which registers all the F# build tasks.
-    See: https://github.com/dotnet/sdk/blob/main/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FSharpTargetsShim.targets
-  -->
-  <PropertyGroup Condition="'$(LoadLocalFSharpBuild)' == 'True'">
     <LocalFSharpBuildBinPath>$(LocalFSharpCompilerPath)/artifacts/bin/fsc/$(LocalFSharpCompilerConfiguration)/$(FSharpNetCoreProductTargetFramework)</LocalFSharpBuildBinPath>
+    <DotnetFscCompilerPath>$(LocalFSharpBuildBinPath)/fsc.dll</DotnetFscCompilerPath>
+    <Fsc_DotNET_DotnetFscCompilerPath>$(LocalFSharpBuildBinPath)/fsc.dll</Fsc_DotNET_DotnetFscCompilerPath>
     <FSharpBuildAssemblyFile>$(LocalFSharpBuildBinPath)/FSharp.Build.dll</FSharpBuildAssemblyFile>
     <FSharpTargetsPath>$(LocalFSharpBuildBinPath)/Microsoft.FSharp.Targets</FSharpTargetsPath>
     <FSharpPropsShim>$(LocalFSharpBuildBinPath)/Microsoft.FSharp.NetSdk.props</FSharpPropsShim>
     <FSharpTargetsShim>$(LocalFSharpBuildBinPath)/Microsoft.FSharp.NetSdk.targets</FSharpTargetsShim>
     <FSharpOverridesTargetsShim>$(LocalFSharpBuildBinPath)/Microsoft.FSharp.Overrides.NetSdk.targets</FSharpOverridesTargetsShim>
+
+    <OtherFlags>$(OtherFlags) --nowarn:75 --times</OtherFlags>
+  </PropertyGroup>
+
+  <!-- Gate 2 (regression matrix only): serve the local FSharp.Core the SDK's own way. FSCorePackageVersion pins the
+       implicit reference; FSCorePackageVersionSet suppresses the SDK default that inits _FSharpCoreLibraryPacksFolder,
+       so we point that (the restore feed) at RegressionLocalCorePackagesDir from the runner, else a co-located library-packs. -->
+  <PropertyGroup Condition="'$(RegressionLocalCore)' == 'true' and '$(FSharpCoreImplicitPackageVersion)' == '' and '$(FSCorePackageVersionSet)' != 'true'">
+    <FSCorePackageVersion>$(RegressionLocalCoreVersion)</FSCorePackageVersion>
+    <FSCorePackageVersionSet>true</FSCorePackageVersionSet>
+    <_FSharpCoreLibraryPacksFolder>$([MSBuild]::ValueOrDefault('$(RegressionLocalCorePackagesDir)', '$(MSBuildThisFileDirectory)library-packs'))</_FSharpCoreLibraryPacksFolder>
   </PropertyGroup>
 
-  <ItemGroup>
+  <Target Name="_ValidateRegressionLocalCore" BeforeTargets="CollectPackageReferences;CoreCompile"
+          Condition="'$(RegressionLocalCore)' == 'true' and '$(RegressionLocalCoreVersion)' == ''">
+    <Error Text="RegressionLocalCore is set but RegressionLocalCoreVersion is empty. The regression job must pass the exact local FSharp.Core package version." />
+  </Target>
+
+  <!-- Gate 3 (CompilerCompat and VS integration): reference the locally built FSharp.Core directly. The regression
+       matrix uses Gate 2's library-packs flow instead, so it is excluded here. -->
+  <PropertyGroup Condition="'$(LoadLocalFSharpBuild)' == 'True' and '$(RegressionLocalCore)' != 'true'">
+    <DisableImplicitFSharpCoreReference>true</DisableImplicitFSharpCoreReference>
+  </PropertyGroup>
+  <ItemGroup Condition="'$(LoadLocalFSharpBuild)' == 'True' and '$(RegressionLocalCore)' != 'true'">
     <Reference Include="$(LocalFSharpCompilerPath)/artifacts/bin/FSharp.Core/$(LocalFSharpCompilerConfiguration)/netstandard2.0/FSharp.Core.dll" />
   </ItemGroup>
 </Project>

UseLocalCompiler.Directory.Build.targets

@@ -0,0 +1,11 @@
+<Project>
+  <!-- Regression matrix only, imported after the project body so these Updates win over the repo's own FSharp.Core
+       reference (the restore feed comes from the .props). CPM needs PackageVersion because a versioned PackageReference
+       is NU1008; Paket and bare Reference repos have nothing to Update and stay on their own core (smoke-only). -->
+  <ItemGroup Condition="'$(RegressionLocalCore)' == 'true' and '$(RegressionLocalCoreVersion)' != '' and '$(ManagePackageVersionsCentrally)' == 'true'">
+    <PackageVersion Update="FSharp.Core" Version="$(RegressionLocalCoreVersion)" />
+  </ItemGroup>
+  <ItemGroup Condition="'$(RegressionLocalCore)' == 'true' and '$(RegressionLocalCoreVersion)' != '' and '$(ManagePackageVersionsCentrally)' != 'true'">
+    <PackageReference Update="FSharp.Core" Version="$(RegressionLocalCoreVersion)" />
+  </ItemGroup>
+</Project>

azure-pipelines-PR.yml

@@ -583,13 +583,19 @@ stages:
             condition: succeeded()
 
           - pwsh: |
-              # Stage UseLocalCompiler props and TargetFrameworks.props together
+              # Stage the props and the locally built FSharp.Core so regression jobs restore it the SDK way.
+              # Arcade packs FSharp.Core into a `Shipping` leaf whose parent varies by layout, so search for it.
               $stagingDir = "$(Build.SourcesDirectory)/UseLocalCompilerPropsStaging"
-              New-Item -ItemType Directory -Force -Path $stagingDir | Out-Null
+              $packsDir = Join-Path $stagingDir "library-packs"
+              New-Item -ItemType Directory -Force -Path $packsDir | Out-Null
               Copy-Item "$(Build.SourcesDirectory)/UseLocalCompiler.Directory.Build.props" -Destination $stagingDir
               Copy-Item "$(Build.SourcesDirectory)/eng/TargetFrameworks.props" -Destination $stagingDir
+              $core = Get-ChildItem "$(Build.SourcesDirectory)/artifacts/packages/Release" -Recurse -Filter "FSharp.Core.*.nupkg" |
+                Where-Object { $_.Name -notlike "*.symbols.nupkg" -and $_.Directory.Name -eq "Shipping" } | Select-Object -First 1
+              if (-not $core) { Write-Host "##[error]FSharp.Core.*.nupkg not found under artifacts/packages/Release/**/Shipping"; exit 1 }
+              Copy-Item $core.FullName -Destination $packsDir
               Write-Host "Staged files for UseLocalCompilerProps artifact:"
-              Get-ChildItem $stagingDir -Name
+              Get-ChildItem $stagingDir -Recurse -Name
             displayName: Stage UseLocalCompiler props files
 
           - task: PublishPipelineArtifact@1
@@ -749,6 +755,7 @@ stages:
           commit: bbe2dec4d0379b5d7d0480997858c30d442fbb42
           buildScript: dotnet build -bl
           displayName: UMX_Slow_Repro        
+          expectLocalCore: true
         - repo: fsprojects/FSharpPlus
           commit: f614035b75922aba41ed6a36c2fc986a2171d2b8
           buildScript: build.cmd
@@ -763,6 +770,7 @@ stages:
           commit: 2648efe
           buildScript: dotnet build tests/FSharpPlus.Tests/FSharpPlus.Tests.fsproj -c Release -bl
           displayName: FsharpPlus_NET10_Build_Lib_Tests
+          expectLocalCore: true
          # remove this before merging
         - repo: fsprojects/FSharpPlus
           commit: 2648efe
@@ -814,10 +822,18 @@ stages:
           commit: 6ddc28d46f81447eacb241b96e16ce693b210c96
           buildScript: dotnet build Prime.sln --configuration Release
           displayName: Prime_Build
+          expectLocalCore: true
         - repo: bryanedds/Nu
           commit: e81e00a464b9d35d272f61708a1a0bfbf487b6d5
           buildScript: dotnet build Nu.sln --configuration Release
           displayName: Nu_Build
+          expectLocalCore: true
+        # Explicit FSharp.Core PackageReference mixed with one implicit test project; the trigger for #20059 (Lanayx/Oxpecker#20065).
+        - repo: Lanayx/Oxpecker
+          commit: cb7e4b83e3f2aba7ded46b17a36d1aea8c13c292
+          buildScript: dotnet build Oxpecker.slnx -c Release
+          displayName: Oxpecker_Build
+          expectLocalCore: true
         # Design-time provider packaging oracle. Pin d8aba70 (pre-workaround base of FSharp.Data.GraphQL#583):
         # it uses the bare IsFSharpDesignTimeProvider gesture, so the client pack drops the provider without this fix.
         # Linux-only: the $PWD local feed and the grep content assertion need bash. (nupkg entry names are stored

docs/regression-testing-pipeline.md

@@ -25,8 +25,8 @@ The regression testing logic is implemented as a reusable Azure DevOps template
 1. **Build F# Compiler**: The `EndToEndBuildTests` job builds the F# compiler and publishes required artifacts
 2. **Matrix Execution**: For each library in the test matrix (running in parallel):
    - Checkout the third-party repository at a specific commit
-   - Install appropriate .NET SDK version using the repository's `global.json`
-   - Setup `Directory.Build.props` to import `UseLocalCompiler.Directory.Build.props`
+   - Pin `global.json` to the exact SDK that built the local compiler
+   - Inject `UseLocalCompiler.Directory.Build.props` via `CustomAfterDirectoryBuildProps`
    - Build the library using its standard build script
    - Publish MSBuild binary logs for analysis
 3. **Report Results**: Success/failure status is reported with build logs for diagnosis
@@ -68,8 +68,9 @@ To add a new library to the test matrix, update the template invocation in `azur
 Each test matrix entry requires:
 - **repo**: GitHub repository in `owner/name` format
 - **commit**: Specific commit SHA for reproducible results
-- **buildScript**: Build script to execute (e.g., `build.cmd`, `build.sh`)
+- **buildScript**: Build command to execute — a `dotnet ...` command or a script file (`build.cmd`/`build.sh`); `;;` separates commands run sequentially, fail-fast
 - **displayName**: Human-readable name for the job
+- **expectLocalCore** (optional): set `true` when the repo has projects that take the implicit `FSharp.Core`; the job then fails unless the locally built FSharp.Core is actually restored — a tripwire for a silently broken shim
 
 ## Pipeline Configuration
 
@@ -82,10 +83,10 @@ Regression tests run automatically as part of PR builds when:
 
 ### Build Environment
 
-- **OS**: Windows (using `$(WindowsMachineQueueName)`)
+- **OS**: Windows by default (`$(WindowsMachineQueueName)`); matrix entries can override to Linux. The scripts are OS-agnostic
 - **Pool**: Standard public build pool (`$(DncEngPublicBuildPool)`)
-- **Timeout**: 60 minutes per regression test job
-- **.NET SDK**: Automatically detects and installs SDK version from each repository's `global.json`
+- **Timeout**: 120 minutes per regression test job
+- **.NET SDK**: Each test repo's `global.json` is pinned to the SDK that built the local compiler, so `fsc.dll` and its host runtime line up
 
 ### Artifacts
 
@@ -108,26 +109,19 @@ When a regression test fails:
 
 ### Local Testing
 
-To test a library locally with your F# compiler build:
+To reproduce a regression locally, on any OS, without editing the library:
 
-1. Build the F# compiler: `.\Build.cmd -c Release -pack`
-
-2. In the third-party library directory, create a `Directory.Build.props`:
-   ```xml
-   <Project>
-     <Import Project="path/to/UseLocalCompiler.Directory.Build.props" />
-   </Project>
+1. Build the compiler and pack FSharp.Core in your `dotnet/fsharp` checkout: `./build.sh -c Release -pack` (`Build.cmd` on Windows).
+2. Clone the library at the failing commit and build it against your local build:
    ```
-
-3. Update the `LocalFSharpCompilerPath` in `UseLocalCompiler.Directory.Build.props` to point to your F# repository.
-
-4. Set environment variables:
-   ```cmd
-   set LoadLocalFSharpBuild=true
-   set LocalFSharpCompilerConfiguration=Release
+   git clone --recursive https://github.com/<owner>/<repo>.git TestRepo
+   cd TestRepo && git checkout <commit>
+   # If TestRepo's global.json pins a different SDK, align sdk.version with <fsharp-repo>/global.json
+   # (allowPrerelease: true, rollForward: disable) — the clone is disposable, as in CI.
+   dotnet fsi <fsharp-repo>/eng/scripts/BuildWithLocalFSharp.fsx --build-script '<the repo build command>'
    ```
 
-5. Run the library's build script.
+`BuildWithLocalFSharp.fsx` runs the same command CI runs, from the current directory, without touching the repo's sources. Add `--verify` to fail unless every project consumes the local FSharp.Core; the script header lists the other options. Because the local package keeps a fixed `-dev` version, the script evicts it from the global NuGet cache before each run so a rebuild is never served stale; pass `--nuget-packages <dir>` to use an isolated cache when running several builds concurrently or against a repo that redirects its packages folder.
 
 ## Best Practices
 
@@ -148,20 +142,7 @@ To test a library locally with your F# compiler build:
 
 ### UseLocalCompiler.Directory.Build.props
 
-This MSBuild props file configures projects to use the locally built F# compiler instead of the SDK version. Key settings:
-
-- `LocalFSharpCompilerPath`: Points to the F# compiler artifacts
-- `DotnetFscCompilerPath`: Path to the fsc.dll compiler
-- `DisableImplicitFSharpCoreReference`: Ensures local FSharp.Core is used
-
-### Path Handling
-
-The pipeline dynamically updates paths in the props file using PowerShell:
-```powershell
-$content -replace 'LocalFSharpCompilerPath.*MSBuildThisFileDirectory.*', 'LocalFSharpCompilerPath>$(Pipeline.Workspace)/FSharpCompiler<'
-```
-
-This ensures the correct path is used in the Azure DevOps environment.
+This MSBuild props file redirects projects to the locally built F# compiler (and, for the matrix, the locally built FSharp.Core) instead of the SDK version. It is organised into gates so it can be injected into unmodified repos as well as imported directly by in-repo tests — see the `Gate 1/2/3` comments in the file. Its companion `UseLocalCompiler.Directory.Build.targets` is injected via `CustomAfterDirectoryBuildTargets` (after the target repo's project body) so the local FSharp.Core version wins over the repo's own reference, whether implicit, an explicit `PackageReference Include`, a `PackageReference Update`, or a central `PackageVersion` (Central Package Management).
 
 ## Future Enhancements