Skip to content

Try using helix sdk support directly again #23585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Jul 22, 2020
Merged

Try using helix sdk support directly again #23585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
99eec40
Use cli support
HaoK Jul 1, 2020
89f4e78
Update runtests.cmd
HaoK Jul 1, 2020
7fe916a
Update InstallDotNet.ps1
HaoK Jul 1, 2020
8a17c83
Update runtests.sh
HaoK Jul 1, 2020
bd61788
Update helix.proj
HaoK Jul 1, 2020
8069366
Update runtests.sh
HaoK Jul 1, 2020
e672b5b
Update ci.yml
HaoK Jul 2, 2020
f8803f4
Update helix.proj
HaoK Jul 2, 2020
32c8bd6
Update helix.proj
HaoK Jul 2, 2020
9afddd5
Update ci.yml
HaoK Jul 2, 2020
47e3e55
Update runtests.cmd
HaoK Jul 3, 2020
234dd7e
Update runtests.sh
HaoK Jul 3, 2020
91bcabf
Update InstallDotNet.ps1
HaoK Jul 3, 2020
89ca365
Update InstallDotNet.ps1
HaoK Jul 3, 2020
f23e13a
Set DOTNET_HOME to dotnet_root
HaoK Jul 6, 2020
b67e31e
Temp no warn to make sure things are good
HaoK Jul 7, 2020
c30bcf6
Remove sdk version arg
HaoK Jul 7, 2020
a228673
Remove sdkVersion
HaoK Jul 7, 2020
923927f
Remove sdk version
HaoK Jul 7, 2020
6c5dd32
Remove sdk version
HaoK Jul 7, 2020
324a900
Update helix.proj
HaoK Jul 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 3 additions & 10 deletions eng/helix/content/InstallDotNet.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
<#
.SYNOPSIS
Installs dotnet sdk and runtime using https://dot.net/v1/dotnet-install.ps1
Installs dotnet runtime using https://dot.net/v1/dotnet-install.ps1
.DESCRIPTION
Installs dotnet sdk and runtime using https://dot.net/v1/dotnet-install.ps1
Installs dotnet runtime using https://dot.net/v1/dotnet-install.ps1
.PARAMETER arch
The architecture to install.
.PARAMETER sdkVersion
The sdk version to install
.PARAMETER runtimeVersion
The runtime version to install
.PARAMETER installDir
Expand All @@ -16,9 +14,6 @@ param(
[Parameter(Mandatory = $true)]
$arch,

[Parameter(Mandatory = $true)]
$sdkVersion,

[Parameter(Mandatory = $true)]
$runtimeVersion,

Expand All @@ -28,8 +23,6 @@ param(

& $PSScriptRoot\Download.ps1 "https://dot.net/v1/dotnet-install.ps1" $PSScriptRoot\dotnet-install.ps1
Write-Host "Download of dotnet-install.ps1 complete..."
Write-Host "Installing SDK...& $PSScriptRoot\dotnet-install.ps1 -Architecture $arch -Version $sdkVersion -InstallDir $installDir"
Invoke-Expression "& $PSScriptRoot\dotnet-install.ps1 -Architecture $arch -Version $sdkVersion -InstallDir $installDir"
Write-Host "Installing Runtime...& $PSScriptRoot\dotnet-install.ps1 -Architecture $arch -Runtime dotnet -Version $runtimeVersion -InstallDir $installDir"
Invoke-Expression "& $PSScriptRoot\dotnet-install.ps1 -Architecture $arch -Runtime dotnet -Version $runtimeVersion -InstallDir $installDir"
Write-Host "InstallDotNet.ps1 complete..."
Write-Host "InstallDotNet.ps1 complete..."
7 changes: 0 additions & 7 deletions eng/helix/content/RunTests/RunTestsOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,6 @@ public static RunTestsOptions Parse(string[] args)
description: "The test dll to run")
{ Argument = new Argument<string>(), Required = true },

new Option(
aliases: new string[] { "--sdk" },
description: "The version of the sdk being used")
{ Argument = new Argument<string>(), Required = true },

new Option(
aliases: new string[] { "--runtime" },
description: "The version of the runtime being used")
Expand Down Expand Up @@ -70,7 +65,6 @@ public static RunTestsOptions Parse(string[] args)
var parseResult = command.Parse(args);
var options = new RunTestsOptions();
options.Target = parseResult.ValueForOption<string>("--target");
options.SdkVersion = parseResult.ValueForOption<string>("--sdk");
options.RuntimeVersion = parseResult.ValueForOption<string>("--runtime");
options.HelixQueue = parseResult.ValueForOption<string>("--queue");
options.Architecture = parseResult.ValueForOption<string>("--arch");
Expand All @@ -86,7 +80,6 @@ public static RunTestsOptions Parse(string[] args)
}

public string Target { get; set;}
public string SdkVersion { get; set;}
public string RuntimeVersion { get; set;}
public string AspNetRuntime { get; set;}
public string AspNetRef { get; set;}
Expand Down
33 changes: 17 additions & 16 deletions eng/helix/content/runtests.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,35 +4,36 @@ setlocal enabledelayedexpansion

REM Use '$' as a variable name prefix to avoid MSBuild variable collisions with these variables
set $target=%1
set $sdkVersion=%2
set $runtimeVersion=%3
set $queue=%4
set $arch=%5
set $quarantined=%6
set $ef=%7
set $aspnetruntime=%8
set $aspnetref=%9
REM Batch only supports up to 9 arguments using the %# syntax, need to shift to get more
shift
set $runtimeVersion=%2
set $queue=%3
set $arch=%4
set $quarantined=%5
set $ef=%6
set $aspnetruntime=%7
set $aspnetref=%8
set $helixTimeout=%9
REM Batch only supports up to 9 arguments using the %# syntax, need to shift to get more

set DOTNET_HOME=%HELIX_CORRELATION_PAYLOAD%\sdk
set DOTNET_ROOT=%DOTNET_HOME%\%$arch%
set DOTNET_ROOT=%HELIX_CORRELATION_PAYLOAD%\dotnet
set DOTNET_HOME=%DOTNET_ROOT%
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In our previous conversations about this change and about incorrectly changing the payload folders, I got the impression we were getting out from under the payload folders with this PR. Won't we hit leftover files still❔

I'm hoping this is fine and that I misunderstood you and @MattGal 😈

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to install the runtime to a different directory we can still do that in a different PR, that's orthogonal to using the sdk support

set DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
set DOTNET_MULTILEVEL_LOOKUP=0
set DOTNET_CLI_HOME=%HELIX_CORRELATION_PAYLOAD%\home

set PATH=%DOTNET_ROOT%;!PATH!;%HELIX_CORRELATION_PAYLOAD%\node\bin
echo Set path to: %PATH%
echo "Invoking InstallDotNet.ps1 %$arch% %$sdkVersion% %$runtimeVersion% %DOTNET_ROOT%"
powershell.exe -NoProfile -ExecutionPolicy unrestricted -file InstallDotNet.ps1 %$arch% %$sdkVersion% %$runtimeVersion% %DOTNET_ROOT%
echo "Invoking InstallDotNet.ps1 %$arch% %$runtimeVersion% %DOTNET_ROOT%"
powershell.exe -NoProfile -ExecutionPolicy unrestricted -file InstallDotNet.ps1 %$arch% %$runtimeVersion% %DOTNET_ROOT%

dotnet --list-sdks
dotnet --list-runtimes

set exit_code=0
echo "Restore: dotnet restore RunTests\RunTests.csproj --source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json --source https://api.nuget.org/v3/index.json --ignore-failed-sources..."
dotnet restore RunTests\RunTests.csproj --source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json --source https://api.nuget.org/v3/index.json --ignore-failed-sources

echo "Running tests: dotnet run --project RunTests\RunTests.csproj -- --target %$target% --sdk %$sdkVersion% --runtime %$runtimeVersion% --queue %$queue% --arch %$arch% --quarantined %$quarantined% --ef %$ef% --aspnetruntime %$aspnetruntime% --aspnetref %$aspnetref% --helixTimeout %$helixTimeout%..."
dotnet run --project RunTests\RunTests.csproj -- --target %$target% --sdk %$sdkVersion% --runtime %$runtimeVersion% --queue %$queue% --arch %$arch% --quarantined %$quarantined% --ef %$ef% --aspnetruntime %$aspnetruntime% --aspnetref %$aspnetref% --helixTimeout %$helixTimeout%
echo "Running tests: dotnet run --project RunTests\RunTests.csproj -- --target %$target% --runtime %$runtimeVersion% --queue %$queue% --arch %$arch% --quarantined %$quarantined% --ef %$ef% --aspnetruntime %$aspnetruntime% --aspnetref %$aspnetref% --helixTimeout %$helixTimeout%..."
dotnet run --project RunTests\RunTests.csproj -- --target %$target% --runtime %$runtimeVersion% --queue %$queue% --arch %$arch% --quarantined %$quarantined% --ef %$ef% --aspnetruntime %$aspnetruntime% --aspnetref %$aspnetref% --helixTimeout %$helixTimeout%
if errorlevel neq 0 (
set exit_code=%errorlevel%
)
Expand Down
26 changes: 7 additions & 19 deletions eng/helix/content/runtests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
#!/usr/bin/env bash

dotnet_sdk_version="$2"
dotnet_runtime_version="$3"
dotnet_runtime_version="$2"

RESET="\033[0m"
RED="\033[0;31m"
Expand All @@ -11,7 +10,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# Ensures every invocation of dotnet apps uses the same dotnet.exe
# Add $random to path to ensure tests don't expect dotnet to be in a particular path
export DOTNET_ROOT="$DIR/.dotnet$RANDOM"
export DOTNET_ROOT="$HELIX_CORRELATION_PAYLOAD/dotnet"

# Ensure dotnet comes first on PATH
export PATH="$DOTNET_ROOT:$PATH:$DIR/node/bin"
Expand Down Expand Up @@ -47,20 +46,6 @@ fi
# Call "sync" between "chmod" and execution to prevent "text file busy" error in Docker (aufs)
chmod +x "dotnet-install.sh"; sync

./dotnet-install.sh --version $dotnet_sdk_version --install-dir "$DOTNET_ROOT"
if [ $? -ne 0 ]; then
sdk_retries=3
while [ $sdk_retries -gt 0 ]; do
./dotnet-install.sh --version $dotnet_sdk_version --install-dir "$DOTNET_ROOT"
if [ $? -ne 0 ]; then
let sdk_retries=sdk_retries-1
echo -e "${YELLOW}Failed to install .NET Core SDK $version. Retries left: $sdk_retries.${RESET}"
else
sdk_retries=0
fi
done
fi

./dotnet-install.sh --runtime dotnet --version $dotnet_runtime_version --install-dir "$DOTNET_ROOT"
if [ $? -ne 0 ]; then
runtime_retries=3
Expand All @@ -85,11 +70,14 @@ fi
# dontet-install.sh seems to affect the Linux filesystem and causes test flakiness unless we sync the filesystem before running tests
sync

$DOTNET_ROOT/dotnet --list-sdks
$DOTNET_ROOT/dotnet --list-runtimes

exit_code=0
echo "Restore: $DOTNET_ROOT/dotnet restore RunTests/RunTests.csproj --source https://api.nuget.org/v3/index.json --ignore-failed-sources..."
$DOTNET_ROOT/dotnet restore RunTests/RunTests.csproj --source https://api.nuget.org/v3/index.json --ignore-failed-sources
echo "Running tests: $DOTNET_ROOT/dotnet run --project RunTests/RunTests.csproj -- --target $1 --sdk $2 --runtime $3 --queue $4 --arch $5 --quarantined $6 --ef $7 --aspnetruntime $8 --aspnetref $9 --helixTimeout ${10}..."
$DOTNET_ROOT/dotnet run --project RunTests/RunTests.csproj -- --target $1 --sdk $2 --runtime $3 --queue $4 --arch $5 --quarantined $6 --ef $7 --aspnetruntime $8 --aspnetref $9 --helixTimeout ${10}
echo "Running tests: $DOTNET_ROOT/dotnet run --project RunTests/RunTests.csproj -- --target $1 --runtime $2 --queue $3 --arch $4 --quarantined $5 --ef $6 --aspnetruntime $7 --aspnetref $8 --helixTimeout $9..."
$DOTNET_ROOT/dotnet run --project RunTests/RunTests.csproj -- --target $1 --runtime $2 --queue $3 --arch $4 --quarantined $5 --ef $6 --aspnetruntime $7 --aspnetref $8 --helixTimeout $9
exit_code=$?
echo "Finished tests...exit_code=$exit_code"

Expand Down
5 changes: 3 additions & 2 deletions eng/helix/helix.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<Project Sdk="Microsoft.DotNet.Helix.Sdk" DefaultTargets="Test" TreatAsLocalProperty="ProjectToBuild">

<PropertyGroup>
<!--
When invoking helix.proj for the whole repo with build.cmd, ProjectToBuild will be set to the path to this project.
Expand All @@ -22,7 +21,9 @@
<IsExternal>true</IsExternal>
<MaxRetryCount Condition="'$(MaxRetryCount)' == ''">2</MaxRetryCount>
<HelixAccessToken Condition="'$(_UseHelixOpenQueues)' != 'true'">$(HelixApiAccessToken)</HelixAccessToken>

<IncludeDotNetCli>true</IncludeDotNetCli>
<DotNetCliPackageType>sdk</DotNetCliPackageType>

<!-- Copied from Microsoft.NET.DefaultOutputPaths.targets in the .NET SDK. The Helix SDK contains nothing similar. -->
<BaseOutputPath Condition="'$(BaseOutputPath)' == ''">bin\</BaseOutputPath>
<OutputPath Condition="'$(OutputPath)' == '' and '$(PlatformName)' == 'AnyCPU'">$(BaseOutputPath)$(Configuration)\</OutputPath>
Expand Down
4 changes: 2 additions & 2 deletions eng/targets/Helix.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,8 @@ Usage: dotnet msbuild /t:Helix src/MyTestProject.csproj
<TestAssembly>$(TargetFileName)</TestAssembly>
<PreCommands>@(HelixPreCommand)</PreCommands>
<PostCommands>@(HelixPostCommand)</PostCommands>
<Command Condition="$(IsWindowsHelixQueue)">call runtests.cmd $(TargetFileName) $(NETCoreSdkVersion) $(MicrosoftNETCoreAppRuntimeVersion) $(_HelixFriendlyNameTargetQueue) $(TargetArchitecture) $(RunQuarantinedTests) $(DotnetEfPackageVersion) Microsoft.AspNetCore.App.Runtime.win-x64.$(AppRuntimeVersion).nupkg Microsoft.AspNetCore.App.Ref.$(AppRuntimeVersion).nupkg $(HelixTimeout)</Command>
<Command Condition="!$(IsWindowsHelixQueue)">./runtests.sh $(TargetFileName) $(NETCoreSdkVersion) $(MicrosoftNETCoreAppRuntimeVersion) $(_HelixFriendlyNameTargetQueue) $(TargetArchitecture) $(RunQuarantinedTests) $(DotnetEfPackageVersion) Microsoft.AspNetCore.App.Runtime.win-x64.$(AppRuntimeVersion).nupkg Microsoft.AspNetCore.App.Ref.$(AppRuntimeVersion).nupkg $(HelixTimeout)</Command>
<Command Condition="$(IsWindowsHelixQueue)">call runtests.cmd $(TargetFileName) $(MicrosoftNETCoreAppRuntimeVersion) $(_HelixFriendlyNameTargetQueue) $(TargetArchitecture) $(RunQuarantinedTests) $(DotnetEfPackageVersion) Microsoft.AspNetCore.App.Runtime.win-x64.$(AppRuntimeVersion).nupkg Microsoft.AspNetCore.App.Ref.$(AppRuntimeVersion).nupkg $(HelixTimeout)</Command>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Typically this would be expressed like:
Condition=" '$(IsWindowsHelixQueue)' == 'true' "> or Condition=" '$(IsWindowsHelixQueue)' != 'true' ">

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MattGal that's true when the property defaults to empty. But, when a property is always set to true or false, using it directly as a bool is easier to read and 💯% supported in msbuild and dotnet msbuild.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aspnetcore/eng/targets/Helix.props

Lines 17 to 18 in 324a900

<IsWindowsHelixQueue>false</IsWindowsHelixQueue>
<IsWindowsHelixQueue Condition="$(HelixTargetQueue.Contains('Windows')) or $(HelixTargetQueue.Contains('windows'))">true</IsWindowsHelixQueue>

<Command Condition="!$(IsWindowsHelixQueue)">./runtests.sh $(TargetFileName) $(MicrosoftNETCoreAppRuntimeVersion) $(_HelixFriendlyNameTargetQueue) $(TargetArchitecture) $(RunQuarantinedTests) $(DotnetEfPackageVersion) Microsoft.AspNetCore.App.Runtime.win-x64.$(AppRuntimeVersion).nupkg Microsoft.AspNetCore.App.Ref.$(AppRuntimeVersion).nupkg $(HelixTimeout)</Command>
<Command Condition="$(HelixCommand) != ''">$(HelixCommand)</Command>
<Timeout>$(HelixTimeout)</Timeout>
</HelixWorkItem>
Expand Down