EnableAzurePipelinesReporter for helix #8094

pakrym · 2019-03-01T18:35:53Z

Fix: #7368
Fix: #8087

analogrelay · 2019-03-01T19:20:03Z

Test Run Aborted... *shakes fist*

analogrelay · 2019-03-01T20:34:29Z

I don't see any results :(

pakrym · 2019-03-01T20:54:05Z

Yep, me neither. Investigating.

HaoK · 2019-03-01T21:53:16Z

adding @MattGal in case there's anything else we might need to enable other than just the flag

pakrym · 2019-03-01T21:56:18Z

I wonder if some of our custom logic might be the cause.

HaoK · 2019-03-01T21:57:16Z

The good news is that it doesn't blow up with the issue I saw when I tried it the first time: #7368

HaoK · 2019-03-01T22:00:34Z

We also don't have xunit reporter turned on, not sure if that is a requirement for the azdo helix stuff

https://github.com/dotnet/arcade/blob/2ec202777efff9fd36518804438f7272a34040f4/eng/common/templates/steps/send-to-helix.yml#L20

pakrym · 2019-03-05T03:06:46Z

@alexperovich any idea why only random small subset of test results got uploaded to VSTS?

eng/helix/vstest/runtests.cmd

eng/helix/vstest/runtests.sh

pakrym · 2019-03-05T04:09:42Z

XUnit results are not published to MC either

HaoK · 2019-03-05T05:49:54Z

Nice so it looks like azdo flags missing test results as an error, so maybe we don't need the explicit test discovery check I just added to our scripts anymore if it fails the run with

E:\A\_work\3\s\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19105.2\tools\azure-pipelines\AzurePipelines.MultiQueue.targets(39,5): error : Test run 2576054 has one or more failing tests. [E:\A\_work\3\s\eng\helix\helix.proj]

pakrym · 2019-03-05T16:18:22Z

Nice so it looks like azdo flags missing test results as an error, so maybe we don't need the explicit test discovery check I just added to our scripts anymore if it fails the run with

It doesn't have reported tests because of your check that prevented the reporter from running.

pakrym · 2019-03-05T16:19:19Z

Tests are not reported because RTX and xUnit XML are not the same.

pakrym · 2019-03-05T16:37:36Z

Blocked dotnet/arcade#2161

natemcmaster · 2019-04-11T22:48:27Z

This should be unblocked now (dotnet/arcade#2286), but @pakrym has switched teams. @JunTaoLuo can you take a look at completing this?

natemcmaster · 2019-04-15T23:10:43Z

Looks good so far. The helix test is still running, but I already see the UI preparing to show results

dougbu · 2019-04-16T00:40:44Z

Even more than the usual number of warnings in the Helix build output. And, it looks like the Helix run caught some flaky tests. Either fix some of the warnings and update this branch or retry AspNetCore-helix-test before we merge.

HaoK · 2019-04-16T22:26:04Z

Hrm helix run looks pretty normal with only interop and one IIS test failing, but the test results all seem to have failed with missing result files, or am I looking at the wrong view?

JunTaoLuo · 2019-04-17T01:40:25Z

I think the warnings are due to dotnet/arcade#1605. I'm not sure there's a fix for it. I'll retry the tests though.

HaoK · 2019-04-30T18:01:28Z

Could it be the additional flaky helix test run that's messing things up when it produces a second trx file? Maybe we can we try removing that additional pass from the runtest.cmd/sh and see if that affects the count?

JunTaoLuo · 2019-04-30T18:07:00Z

Hmm this seems suspicious in the log:

Found results file /Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Work/3794520e-9043-4e3c-bf3d-120e76f5d6ef/Exec/TestResults/_dci-mac-build-080_2019-04-30_01_55_20_332.trx with format trx
Uploading results in batches of size 1000
Traceback (most recent call last):
  File "/Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Payload/reporter/run.py", line 95, in <module>
    main()
  File "/Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Payload/reporter/run.py", line 84, in main
    for b in batches:
  File "/Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Payload/reporter/helpers.py", line 13, in batch
    for item in iterable:
  File "/Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Payload/reporter/test_results_reader/__init__.py", line 94, in <genexpr>
    return (add_logs(tr, log_list) for tr in f.read_results(file_path))
  File "/Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Payload/reporter/formats/trx.py", line 41, in read_results
    classname = test_classes[test_name]
KeyError: 'Microsoft.AspNetCore.Mvc.ApplicationModels.AttributeRouteModelTests.ReplaceTokens_InvalidFormat(template: "text]", values: [], reason: "Token delimiters (\'[\', \']\') are imbalanced.")'
+ date -u +%FT%TZ
2019-04-30T08:55:31Z
+ /usr/local/bin/python2 /Users/dotnet-bot/dotnetbuild/work/4bc49cfb-4e54-4e80-b956-4085f63855a8/Payload/xunit-reporter.py
2019-04-30 01:55:32,174: ERROR: xunit-reporter(69): main: Unable to report xunit results: no test results xml file found.

Maybe the uploading of the results went awry.

alexperovich · 2019-04-30T18:38:11Z

Ugh, thats just a bug with the trx parsing. Trx is sooo complicated for a simple type of data. I'll fix it.

HaoK · 2019-04-30T18:40:54Z

Cool, that's good news, well hopefully its just that bug on the parsing, and all our test data is already there

alexperovich · 2019-04-30T19:07:16Z

PR with fix: dotnet/arcade#2648

JunTaoLuo · 2019-04-30T21:35:33Z

@alexperovich can you let me know which version of the sdk will have this fix when a build of it is available?

alexperovich · 2019-04-30T21:39:34Z

The build is currently waiting behind another build. I will let you know when a version containing the fix is available.

alexperovich · 2019-04-30T22:27:07Z

Version 2.0.0-beta.19230.6 has been published with the fix.

JunTaoLuo · 2019-05-01T19:45:48Z

Still missing some test results as Helix is only reporting from one of the .trx files. Will wait for another SDK update.

JunTaoLuo · 2019-05-03T19:16:22Z

Woot! I think all the test results are being reported now, thanks @alexperovich for the many fixes. I think after a final round of tests, this will be ready to get checked in!

Test reporting is fixed

JunTaoLuo · 2019-05-06T00:13:53Z

src/Servers/Kestrel/test/FunctionalTests/Http2/ShutdownTests.cs

@@ -23,6 +23,7 @@ namespace Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.Http2
 {
    [OSSkipCondition(OperatingSystems.MacOSX, SkipReason = "Missing SslStream ALPN support: https://github.com/dotnet/corefx/issues/30492")]
    [MinimumOSVersion(OperatingSystems.Windows, WindowsVersions.Win10)]
+    [SkipOnHelix(Queues = "Debian.8.Amd64.Open")] // Debian 8 uses OpenSSL 1.0.1 which does not support HTTP/2


Note to self: double check the versino of OpenSSL on Debian 8 agents and see if there's a way to manually update the version to one that supports HTTP/2

Eilon added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Mar 3, 2019

pakrym mentioned this pull request Mar 4, 2019

Helix fail if exceptions thrown during test discovery #8107

Merged

pakrym marked this pull request as ready for review March 4, 2019 22:19

pakrym requested a review from dougbu as a code owner March 4, 2019 22:19

pakrym requested review from HaoK and analogrelay March 4, 2019 22:24

dougbu reviewed Mar 5, 2019

View reviewed changes

eng/helix/vstest/runtests.cmd Outdated Show resolved Hide resolved

eng/helix/vstest/runtests.sh Outdated Show resolved Hide resolved

analogrelay mentioned this pull request Mar 5, 2019

Client reconnects when state's available on the server #7395

Merged

pakrym added the blocked The work on this issue is blocked due to some dependency label Mar 5, 2019

natemcmaster assigned JunTaoLuo Apr 11, 2019

JunTaoLuo force-pushed the pakrym/EnableAzurePipelinesReporter branch from 2f00d10 to 5102e55 Compare April 15, 2019 22:30

pakrym requested a review from a team as a code owner April 15, 2019 22:30

JunTaoLuo force-pushed the pakrym/EnableAzurePipelinesReporter branch from 28dfffc to 8c2a5bb Compare May 1, 2019 07:30

EnableAzurePipelinesReporter for helix

ad66401

JunTaoLuo force-pushed the pakrym/EnableAzurePipelinesReporter branch from 974d43a to ad66401 Compare May 2, 2019 21:13

John Luo added 3 commits May 3, 2019 10:15

Update Helix SDK

787f8d9

Merge branch 'master' into pakrym/EnableAzurePipelinesReporter

e59388a

Revert a fix that broke things even more

d778c48

natemcmaster approved these changes May 3, 2019

View reviewed changes

John Luo added 9 commits May 3, 2019 16:36

Try publishing test assets

671cac2

Try fixing Kestrel tests on Helix

e30aa2c

Try a temp file in the working directory

26d36a5

Skip Http2Spec tests on Debian 8

54d2ef1

Missed update

2733e14

Try a different format

30bcf1b

Merge branch 'master' into pakrym/EnableAzurePipelinesReporter

d2323e0

Disable failing test on Fedora

be38ed9

Mark flaky test

2acbf95

JunTaoLuo reviewed May 6, 2019

View reviewed changes

JunTaoLuo merged commit fa239f2 into master May 6, 2019

ghost deleted the pakrym/EnableAzurePipelinesReporter branch May 6, 2019 05:15

EnableAzurePipelinesReporter for helix #8094

EnableAzurePipelinesReporter for helix #8094

Uh oh!

Conversation

pakrym commented Mar 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

analogrelay commented Mar 1, 2019

Uh oh!

analogrelay commented Mar 1, 2019

Uh oh!

pakrym commented Mar 1, 2019

Uh oh!

HaoK commented Mar 1, 2019

Uh oh!

pakrym commented Mar 1, 2019

Uh oh!

HaoK commented Mar 1, 2019

Uh oh!

HaoK commented Mar 1, 2019

Uh oh!

pakrym commented Mar 5, 2019

Uh oh!

Uh oh!

Uh oh!

pakrym commented Mar 5, 2019

Uh oh!

HaoK commented Mar 5, 2019

Uh oh!

pakrym commented Mar 5, 2019

Uh oh!

pakrym commented Mar 5, 2019

Uh oh!

pakrym commented Mar 5, 2019

Uh oh!

natemcmaster commented Apr 11, 2019

Uh oh!

natemcmaster commented Apr 15, 2019

Uh oh!

dougbu commented Apr 16, 2019

Uh oh!

HaoK commented Apr 16, 2019

Uh oh!

JunTaoLuo commented Apr 17, 2019

Uh oh!

HaoK commented Apr 30, 2019

Uh oh!

JunTaoLuo commented Apr 30, 2019

Uh oh!

alexperovich commented Apr 30, 2019

Uh oh!

HaoK commented Apr 30, 2019

Uh oh!

alexperovich commented Apr 30, 2019

Uh oh!

JunTaoLuo commented Apr 30, 2019

Uh oh!

alexperovich commented Apr 30, 2019

Uh oh!

alexperovich commented Apr 30, 2019

Uh oh!

JunTaoLuo commented May 1, 2019

Uh oh!

JunTaoLuo commented May 3, 2019

Uh oh!

JunTaoLuo May 6, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pakrym commented Mar 1, 2019 •

edited

Loading