Skip to content

Enable compiling with crossgen2 #31778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

Enable compiling with crossgen2 #31778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
34ea78a
Enable compiling Asp.NET with crossgen2
davidwrighton Apr 13, 2021
04fbd5c
Use use linux crossgen2 instead of linux-musl crossgen2 when targetti…
davidwrighton Apr 19, 2021
deb3c7d
Attempt to make musl work
davidwrighton Apr 19, 2021
6f32925
Fix typo
davidwrighton Apr 19, 2021
6348300
Use correct bits to compile take #3
davidwrighton Apr 19, 2021
4232e63
Merge remote-tracking branch 'upstream/main' into Crossgen2Support
davidwrighton Apr 22, 2021
687b77d
Add Pgo data
davidwrighton Apr 22, 2021
2c399f8
Fix mistaken at symbol
davidwrighton Apr 27, 2021
4be922e
Merge branch 'main' of github.com:dotnet/aspnetcore into Crossgen2Sup…
davidwrighton Apr 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .azure/pipelines/quarantined-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
- script: ./eng/build.cmd -ci -nobl -noBuildRepoTasks -noRestore -test -all -noBuildJava -noBuildNative
-projects eng\helix\helix.proj /p:RunQuarantinedTests=true /p:IsRequiredCheck=true /p:IsHelixJob=true
/p:BuildInteropProjects=true /p:RunTemplateTests=true
/p:CrossgenOutput=false /p:ASPNETCORE_TEST_LOG_DIR=artifacts/log
/p:ASPNETCORE_TEST_LOG_DIR=artifacts/log
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a particularly interesting change given we expect failures in the quarantined runs. Suggest instead removing /p:CrossgenOutput=false from the 'Tests: Helix x64' job's build steps in ci.yml. If the crossgen2'd assemblies are platform-specific with the command arguments you're currently using, I'd expect to see a lot of failures on Linux and macOS work items (ProjectTemplates.Test in particular) submitted from that job without the property setting.

displayName: Run build.cmd helix target
continueOnError: true
env:
Expand Down
10 changes: 9 additions & 1 deletion eng/Dependencies.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,14 @@ and are generated based on the last package release.
<LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-musl-x64" />
<LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-musl-arm" />
<LatestPackageReference Include="Microsoft.NETCore.App.Runtime.linux-musl-arm64" />

<!-- Crossgen2 compiler -->
<LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.osx-x64" />
<LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.osx-arm64" />
<LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.linux-musl-x64" />
<LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.linux-x64" />
<LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.win-x64" />
<LatestPackageReference Include="Microsoft.NETCore.App.Crossgen2.win-arm64" />
</ItemGroup>

<ItemGroup Label=".NET team dependencies (Non-source-build)" Condition="'$(DotNetBuildFromSource)' != 'true'">
Expand Down Expand Up @@ -206,7 +214,7 @@ and are generated based on the last package release.
-->
<LatestPackageReference Update="@(LatestPackageReference->WithMetadataValue('Version', ''))">
<Version
Condition=" $([System.String]::new('%(Identity)').StartsWith('Microsoft.NETCore.App.Runtime.')) ">$(MicrosoftNETCoreAppRuntimeVersion)</Version>
Condition=" $([System.String]::new('%(Identity)').StartsWith('Microsoft.NETCore.App.Runtime.')) or $([System.String]::new('%(Identity)').StartsWith('Microsoft.NETCore.App.Crossgen2.'))">$(MicrosoftNETCoreAppRuntimeVersion)</Version>
</LatestPackageReference>
</ItemGroup>
</Project>
60 changes: 55 additions & 5 deletions src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ This package is an internal implementation of the .NET Core SDK and is not meant

<!-- Optimize the framework using the crossgen tool -->
<CrossgenOutput Condition=" '$(CrossgenOutput)' == '' AND '$(Configuration)' != 'Debug' ">true</CrossgenOutput>
<UseCrossgen2>true</UseCrossgen2>

<!-- Produce crossgen profiling symbols (.ni.pdb or .map files). -->
<GenerateCrossgenProfilingSymbols>true</GenerateCrossgenProfilingSymbols>
Expand Down Expand Up @@ -100,6 +101,8 @@ This package is an internal implementation of the .NET Core SDK and is not meant
<!-- Crossgen executable name -->
<CrossgenToolFileName>crossgen</CrossgenToolFileName>
<CrossgenToolFileName Condition=" '$(TargetOsName)' == 'win' ">$(CrossgenToolFileName).exe</CrossgenToolFileName>
<Crossgen2ToolFileName>crossgen2</Crossgen2ToolFileName>
<Crossgen2ToolFileName Condition=" '$(TargetOsName)' == 'win' ">$(Crossgen2ToolFileName).exe</Crossgen2ToolFileName>
<!-- Default crossgen executable relative path -->
<CrossgenToolPackagePath>$(CrossgenToolFileName)</CrossgenToolPackagePath>
<!-- Disambiguated RID-specific crossgen executable relative path -->
Expand All @@ -108,6 +111,11 @@ This package is an internal implementation of the .NET Core SDK and is not meant
<!-- E.g. "PkgMicrosoft_NETCore_App_Runtime_win-x64" (set in obj/Microsoft.AspNetCore.App.Runtime.csproj.nuget.g.props). -->
<RuntimePackageRootVariableName>PkgMicrosoft_NETCore_App_Runtime_$(RuntimeIdentifier)</RuntimePackageRootVariableName>

<BuildOsName>$(TargetOsName)</BuildOsName>
<!-- linux-musl-arm and linux-musl-arm64 are built on an Ubuntu container with cross compilation tools. linux-musl-x64 is built in an alpine container -->
<BuildOsName Condition="'$(TargetOsName)' == 'linux-musl' and '$(TargetArchitecture)'!='x64'">linux</BuildOsName>
<Crossgen2PackageRootVariableName>PkgMicrosoft_NETCore_App_Crossgen2_$(BuildOsName)-$(BuildArchitecture)</Crossgen2PackageRootVariableName>

<AssetTargetFallback>$(AssetTargetFallback);native,Version=0.0</AssetTargetFallback>

<NativePlatform>$(TargetArchitecture)</NativePlatform>
Expand All @@ -125,6 +133,11 @@ This package is an internal implementation of the .NET Core SDK and is not meant
PrivateAssets="All"
GeneratePathProperty="true" />

<Reference Include="Microsoft.NETCore.App.Crossgen2.$(BuildOsName)-$(BuildArchitecture)"
ExcludeAssets="All"
PrivateAssets="All"
GeneratePathProperty="true" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest Microsoft.NETCore.App.Crossgen2.* packages should contain a same-named .props file that sets a property for the tool path. This didn't make sense for Microsoft.NETCore.App.Runtime.* packages but definitely does now. Wouldn't need the path property nor some of the later complications finding the tool if package were more self-describing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any possibility of removing the need for workarounds (hacks) like https://github.com/dotnet/aspnetcore/pull/31778/files#diff-eb8fd31a7971d712606a187ea074d0bd5d6ea85aae5a707107fd998216986ea5R117, https://github.com/dotnet/aspnetcore/pull/31778/files#diff-eb8fd31a7971d712606a187ea074d0bd5d6ea85aae5a707107fd998216986ea5R365-R368, and related bits in the future❔


<ProjectReference Condition=" '$(BuildIisNativeProjects)' == 'true' AND $(BuildNative) "
Include="$(RepoRoot)src\Servers\IIS\AspNetCoreModuleV2\InProcessRequestHandler\InProcessRequestHandler.vcxproj">
<SetPlatform>Platform=$(NativePlatform)</SetPlatform>
Expand Down Expand Up @@ -349,10 +362,16 @@ This package is an internal implementation of the .NET Core SDK and is not meant
<RuntimePackageRootMapping>
<RuntimePackageRoot>$(%(Identity))</RuntimePackageRoot>
</RuntimePackageRootMapping>
<Crossgen2PackageRootMapping Include="$(Crossgen2PackageRootVariableName)" />
<Crossgen2PackageRootMapping>
<Crossgen2PackageRoot>$(%(Identity))</Crossgen2PackageRoot>
</Crossgen2PackageRootMapping>
</ItemGroup>
<PropertyGroup>
<RuntimePackageRoot>@(RuntimePackageRootMapping->'%(RuntimePackageRoot)')</RuntimePackageRoot>
<RuntimePackageRoot>$([MSBuild]::EnsureTrailingSlash('$(RuntimePackageRoot)'))</RuntimePackageRoot>
<Crossgen2PackageRoot>@(Crossgen2PackageRootMapping->'%(Crossgen2PackageRoot)')</Crossgen2PackageRoot>
<Crossgen2PackageRoot>$([MSBuild]::EnsureTrailingSlash('$(Crossgen2PackageRoot)'))</Crossgen2PackageRoot>
</PropertyGroup>
</Target>

Expand All @@ -375,10 +394,18 @@ This package is an internal implementation of the .NET Core SDK and is not meant

<ReferenceCopyLocalPaths Remove="@(IntermediateCrossgenAssembly)" />
<ReferenceCopyLocalPaths Include="@(IntermediateCrossgenAssembly->'$(TargetDir)%(FileName)%(Extension)')" />

<_DistinctPlatformAssemblyPaths Include="@(_PlatformAssemblyPaths->Distinct())"/>
<_DistinctPlatformAssemblyPaths>
<CommandLineOption>-r:%(Identity)*.dll</CommandLineOption>
</_DistinctPlatformAssemblyPaths>

<Crossgen2PlatformAssemblyPaths Include="@(_DistinctPlatformAssemblyPaths->'%(CommandLineOption)')" />
</ItemGroup>

<PropertyGroup>
<CrossgenToolPath>$([System.IO.Path]::Combine('$(RuntimePackageRoot)', 'tools', '$(CrossgenToolPackagePath)'))</CrossgenToolPath>
<CrossgenToolPath Condition="'$(UseCrossgen2)' != 'true'" >$([System.IO.Path]::Combine('$(RuntimePackageRoot)', 'tools', '$(CrossgenToolPackagePath)'))</CrossgenToolPath>
<CrossgenToolPath Condition="'$(UseCrossgen2)' == 'true'" >$([System.IO.Path]::Combine('$(Crossgen2PackageRoot)', 'tools', '$(Crossgen2ToolFileName)'))</CrossgenToolPath>
<CrossgenSymbolsTargetDir>$(TargetDir)</CrossgenSymbolsTargetDir>
<CrossgenPlatformAssemblyPaths>@(_PlatformAssemblyPaths->Distinct(), '$(PathSeparator)')</CrossgenPlatformAssemblyPaths>

Expand All @@ -392,15 +419,21 @@ This package is an internal implementation of the .NET Core SDK and is not meant
File="$(CrossgenToolDir)PlatformAssembliesPaths.rsp"
Overwrite="true" />

<WriteLinesToFile
Lines="@(Crossgen2PlatformAssemblyPaths)"
File="$(CrossgenToolDir)PlatformAssembliesPathsCrossgen2.rsp"
Overwrite="true" />

<ItemGroup>
<RuntimePackageFiles Include="$(RuntimePackageRoot)runtimes\**\*" />
</ItemGroup>

<Error Text="Could not find crossgen $(CrossgenToolPath)" Condition=" ! Exists($(CrossgenToolPath))" />

<!-- Create tool directory with crossgen executable and runtime assemblies -->
<Copy SourceFiles="@(RuntimePackageFiles);$(CrossgenToolPath)" DestinationFolder="$(CrossGenToolDir)"/>
<Exec Command="chmod +x &quot;$(CrossGenToolDir)$(CrossgenToolFileName)&quot;" Condition="'$(OS)' != 'Windows_NT'" />
<Copy SourceFiles="@(RuntimePackageFiles)" DestinationFolder="$(CrossGenToolDir)"/>
<Copy Condition="'$(UseCrossgen2)' != 'true'" SourceFiles="$(CrossgenToolPath)" DestinationFolder="$(CrossGenToolDir)"/>
<Exec Command="chmod +x &quot;$(CrossGenToolDir)$(CrossgenToolFileName)&quot;" Condition="'$(OS)' != 'Windows_NT' and '$(UseCrossgen2)' != 'true'" />
</Target>

<Target Name="_BatchCrossGenAssemblies"
Expand All @@ -413,18 +446,35 @@ This package is an internal implementation of the .NET Core SDK and is not meant
Condition="'$(CrossCompileDirectory)' == ''">$(RuntimePackageRoot)runtimes\$(RuntimeIdentifier)\native\$(LibPrefix)clrjit$(LibExtension)</CoreCLRJitPath>
<CoreCLRJitPath
Condition="'$(CrossCompileDirectory)' != ''">$(RuntimePackageRoot)runtimes\$(CrossCompileDirectory)\native\$(LibPrefix)clrjit$(LibExtension)</CoreCLRJitPath>

<Crossgen2TargetOs>$(TargetOsName)</Crossgen2TargetOs>
<Crossgen2TargetOs Condition="'$(TargetOsName)' == 'win'">windows</Crossgen2TargetOs>
<Crossgen2TargetOs Condition="'$(TargetOsName)' == 'linux-musl'">linux</Crossgen2TargetOs>
<Crossgen2Args>--targetarch:$(TargetArchitecture)</Crossgen2Args>
<Crossgen2Args>$(Crossgen2Args) --targetos:$(Crossgen2TargetOs)</Crossgen2Args>
<Crossgen2Args>$(Crossgen2Args) -O</Crossgen2Args>
<Crossgen2Args>$(Crossgen2Args) @&quot;$(CrossgenToolDir)PlatformAssembliesPathsCrossgen2.rsp&quot;</Crossgen2Args>
<Crossgen2Args Condition="Exists('$(RuntimePackageRoot)tools\StandardOptimizationData.mibc')">$(Crossgen2Args) &quot;-m:$(RuntimePackageRoot)tools\StandardOptimizationData.mibc&quot;</Crossgen2Args>
<Crossgen2Args Condition="'$(GenerateCrossgenProfilingSymbols)' == 'true' and '$(TargetOsName)' == 'win'">$(Crossgen2Args) --pdb --pdb-path:&quot;$(CrossgenSymbolsTargetDir)&quot;</Crossgen2Args>
<Crossgen2Args Condition="'$(GenerateCrossgenProfilingSymbols)' == 'true' and '$(TargetOsName)' != 'win'">$(Crossgen2Args) --perfmap --perfmap-path:&quot;$(CrossgenSymbolsTargetDir)&quot;</Crossgen2Args>
</PropertyGroup>

<Exec Command="&quot;$(CrossgenToolDir)$(CrossgenToolFileName)&quot; -nologo -readytorun -in &quot;%(IntermediateCrossgenAssembly.Identity)&quot; -out &quot;$(TargetDir)%(FileName)%(Extension)&quot; -jitpath &quot;$(CoreCLRJitPath)&quot; &quot;@$(CrossgenToolDir)PlatformAssembliesPaths.rsp&quot;"
<Exec Condition="'$(UseCrossgen2)' != 'true'"
Command="&quot;$(CrossgenToolDir)$(CrossgenToolFileName)&quot; -nologo -readytorun -in &quot;%(IntermediateCrossgenAssembly.Identity)&quot; -out &quot;$(TargetDir)%(FileName)%(Extension)&quot; -jitpath &quot;$(CoreCLRJitPath)&quot; &quot;@$(CrossgenToolDir)PlatformAssembliesPaths.rsp&quot;"
EnvironmentVariables="COMPlus_PartialNGen=0"
IgnoreStandardErrorWarningFormat="true"
StandardOutputImportance="High" />

<Exec Condition=" '$(GenerateCrossgenProfilingSymbols)' == 'true' "
<Exec Condition=" '$(GenerateCrossgenProfilingSymbols)' == 'true' and '$(UseCrossgen2)' != 'true'"
Command="&quot;$(CrossgenToolDir)$(CrossgenToolFileName)&quot; -nologo -readytorun -in &quot;$(TargetDir)%(IntermediateCrossgenAssembly.FileName)%(Extension)&quot; -Create$(CrossGenSymbolsType) &quot;$(CrossgenSymbolsTargetDir)&quot; &quot;@$(CrossgenToolDir)PlatformAssembliesPaths.rsp&quot;"
EnvironmentVariables="COMPlus_PartialNGen=0"
IgnoreStandardErrorWarningFormat="true"
StandardOutputImportance="High" />

<Exec Condition="'$(UseCrossgen2)' == 'true'"
Command="&quot;$(CrossgenToolPath)&quot; $(Crossgen2Args) -o:&quot;$(TargetDir)%(FileName)%(Extension)&quot; &quot;%(IntermediateCrossgenAssembly.Identity)&quot;"
IgnoreStandardErrorWarningFormat="true"
StandardOutputImportance="High" />
</Target>

<!--
Expand Down