Skip to content

Try using xml logger #18491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jan 24, 2020
Merged

Try using xml logger #18491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
36d0f1a
Try using xml logger
HaoK Jan 21, 2020
686b448
Switch to xunit logger
HaoK Jan 21, 2020
1c0c33f
Switch to xunit
HaoK Jan 21, 2020
e65b538
Update Helix.targets
HaoK Jan 22, 2020
630cf54
Make a test fail
HaoK Jan 22, 2020
f68c00b
Copy test results to root
HaoK Jan 23, 2020
1f12480
Try specifying file name
HaoK Jan 24, 2020
8b0ebfe
Just copy test results file explicitly
HaoK Jan 24, 2020
2d77972
Copy test results file explicitly
HaoK Jan 24, 2020
9c8cad9
Remove windows log copying for now
HaoK Jan 24, 2020
aeb75f4
Revert test change
HaoK Jan 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion eng/helix/content/runtests.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ REM only puts the explicit filter traits the user provided in
REM Filter syntax: https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md
set NONFLAKY_FILTER="Flaky:All!=true&Flaky:Helix:All!=true&Flaky:Helix:Queue:All!=true&Flaky:Helix:Queue:%HELIX%!=true"
echo Running non-flaky tests.
%DOTNET_ROOT%\dotnet vstest %target% --logger:trx --TestCaseFilter:%NONFLAKY_FILTER%
%DOTNET_ROOT%\dotnet vstest %target% --logger:xunit --TestCaseFilter:%NONFLAKY_FILTER%
if errorlevel 1 (
echo Failure in non-flaky test 1>&2
set exit_code=1
Expand All @@ -59,5 +59,8 @@ if errorlevel 1 (
REM DO NOT EXIT and DO NOT SET EXIT_CODE to 1
)

echo "Copying TestResults\TestResults.xml to ."
copy TestResults\TestResults.xml testResults.xml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, this means we're only going to get data from un-quarantined tests. We should come up with a plan for how to get this data out. Maybe we have to rethink our quarantine process.

If we ran a single run of all tests, we'd still be able to tell if they were quarantined or not by the traits. Unfortunately, the run would fail even when only quarantined tests fail.

Another option is to change how the quarantine process works and just skip tests that are quarantined. It would mean we can't monitor them for when they get non-flaky, but we'd still be able to track which tests were skipped because they were flaky and track the number of skipped flaky tests over time so that teams can drive that number down.

Food for thought at the next DoI meeting :).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should probably just add a flaky-test.yml and just run all tests in that run every day on master, and get rid of the flaky test pass in our normal PRs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we'd have all the data in kusto

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I think a separate run is the solution. I think we can (and should) continue running them on PRs if we can though. The main downside is that for projects with no quarantined tests, we’ll submit a Helix work item with almost no action, it will run 0 tests. We could start with that though. I did some spelunking and it’s definitely feasible to run a little code to check a test assembly (once built) for any quarantined tests using the vstest APIs. That way we could save submitting a no-op job to helix.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool maybe we can just repurpose the old helix-test pipeline to be the flaky helix test run


exit /b %exit_code%

5 changes: 3 additions & 2 deletions eng/helix/content/runtests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
# Used by SkipOnHelix attribute
export helix="$helix_queue_name"


RESET="\033[0m"
RED="\033[0;31m"
YELLOW="\033[0;33m"
Expand Down Expand Up @@ -94,7 +93,7 @@ fi
# Filter syntax: https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md
NONFLAKY_FILTER="Flaky:All!=true&Flaky:Helix:All!=true&Flaky:Helix:Queue:All!=true&Flaky:Helix:Queue:$helix_queue_name!=true"
echo "Running non-flaky tests."
$DOTNET_ROOT/dotnet vstest $test_binary_path --logger:trx --TestCaseFilter:"$NONFLAKY_FILTER"
$DOTNET_ROOT/dotnet vstest $test_binary_path --logger:xunit --TestCaseFilter:"$NONFLAKY_FILTER"
nonflaky_exitcode=$?
if [ $nonflaky_exitcode != 0 ]; then
echo "Non-flaky tests failed!" 1>&2
Expand All @@ -109,6 +108,8 @@ if [ $? != 0 ]; then
# DO NOT EXIT
fi

echo "Copying TestResults/TestResults to ."
cp TestResults/TestResults.xml testResults.xml
echo "Copying artifacts/logs to $HELIX_WORKITEM_UPLOAD_ROOT/../"
shopt -s globstar
cp artifacts/log/**/*.log $HELIX_WORKITEM_UPLOAD_ROOT/../
Expand Down
17 changes: 11 additions & 6 deletions eng/targets/Helix.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
<Project>

<!-- Item group has to be defined here becasue Helix.props is evaluated before xunit.runner.console.props -->
<ItemGroup Condition="'$(TargetFrameworkIdentifier)' == '.NETFramework'">
<HelixContent Condition="'$(XunitConsole472Path)' != ''" Include="$([System.IO.Path]::GetDirectoryName('$(XunitConsole472Path)'))/**/*" />
</ItemGroup>

<ItemGroup>
<ItemGroup Condition="$(BuildHelixPayload)">
<Content Include="@(HelixContent)" />
</ItemGroup>

<!--

This target is meant to be used when invoking helix tests on one project at a time.
Expand All @@ -35,6 +31,10 @@ Usage: dotnet msbuild /t:Helix src/MyTestProject.csproj
</MSBuild>
</Target>

<ItemGroup>
<PackageReference Include="XunitXml.TestLogger" Version="2.1.26" />
</ItemGroup>

<Target Name="_SetCreateHelixPayload">
<PropertyGroup>
<_SelectedPlatforms>@(HelixProjectPlatform)</_SelectedPlatforms>
Expand All @@ -60,6 +60,11 @@ Usage: dotnet msbuild /t:Helix src/MyTestProject.csproj

<Target Name="_CreateHelixWorkItem" Condition="$(BuildHelixPayload)">

<ItemGroup>
<HelixContent Include="$(OutputPath)/Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestAdapter.dll" />
<HelixContent Include="$(OutputPath)/Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.dll" />
</ItemGroup>

<PropertyGroup>
<!-- Extract the effective queue name from this format "(name)host@dockerimage". This is sometimes used in test code to skip tests. -->
<_HelixFriendlyNameTargetQueue>$(HelixTargetQueue)</_HelixFriendlyNameTargetQueue>
Expand Down