Skip to content

Improve crash/hang support in helix #24489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 1, 2020
Merged

Improve crash/hang support in helix #24489

merged 1 commit into from
Aug 1, 2020

Conversation

HaoK
Copy link
Member

@HaoK HaoK commented Jul 31, 2020

Switch to test to be able to opt into better blame support which improves crash/dump handling, add additional gathering of dumps for upload

@ghost ghost added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Jul 31, 2020
@HaoK HaoK requested review from davidfowl and a team July 31, 2020 20:21
@davidfowl
Copy link
Member

cc @javiercn

Copy link
Contributor

@dougbu dougbu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest adding a Task.Delay(TimeSpan.FromMinutes(7)) in a random async test method (that runs on Helix of course 😈) to confirm this works as expected

Console.WriteLine($"Copying: {file} to {Path.Combine(HELIX_WORKITEM_UPLOAD_ROOT, fileName)}");
// Need to copy to HELIX_WORKITEM_UPLOAD_ROOT and HELIX_WORKITEM_UPLOAD_ROOT/../ in order for Azure Devops attachments to link properly and for Helix to store the logs
File.Copy(file, Path.Combine(HELIX_WORKITEM_UPLOAD_ROOT, fileName));
File.Copy(file, Path.Combine(HELIX_WORKITEM_UPLOAD_ROOT, "..", fileName));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't this requirement a point-in-time thing that's since been resolved❔ @MattGal were you involved in the previous conversations around this❔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dougbu I was OOF Friday but yes, both XUnit and AzDO reporters should now allow this since May or so, via dotnet/arcade#5521 . If the files are small it doesn't really matter, if they're big it could mean a decent perf gain; either way when it's convenient it'd be good to remove this anyways.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok cool, I'll try removing the extra copy to the parent

@@ -247,7 +247,7 @@ public async Task<int> RunTestsAsync()
{
// Timeout test run 5 minutes before the Helix job would timeout
var cts = new CancellationTokenSource(Options.Timeout.Subtract(TimeSpan.FromMinutes(5)));
var commonTestArgs = $"vstest {Options.Target} --logger:xunit --logger:\"console;verbosity=normal\" --blame";
var commonTestArgs = $"test {Options.Target} --logger:xunit --logger:\"console;verbosity=normal\" --blame \"CollectHangDump;TestTimeout=5m\"";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, doesn't vstest expect a project or solution file❔ I'm hoping there's no need to change the subcommand.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note these options aren't particularly well-documented for either dotnet test or dotnet vstest. Might be more correct to use --blame --blame-hang --blame-hang-timeout 5m but it's hard to tell.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts @nohwnd as he's the expert on these new options

@HaoK
Copy link
Member Author

HaoK commented Aug 1, 2020

Yeah should have mentioned this is the result of #23862 which has examples of what hangs and crashes look like now

Copy link
Member

@davidfowl davidfowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ViktorHofer you were saying we should use a RunSettings file? @nohwnd do you have any suggestions here? I'm ok with that approach as long as we're not hardcoding physical paths to anything 👍🏾

@HaoK
Copy link
Member Author

HaoK commented Aug 1, 2020

I'm going to do the runtest settings in a separate PR

@HaoK HaoK merged commit 130ec43 into master Aug 1, 2020
@HaoK HaoK deleted the helix/blame branch August 1, 2020 17:38
@HaoK
Copy link
Member Author

HaoK commented Aug 1, 2020

Followup PR for runsettings here: #24503

@ViktorHofer
Copy link
Member

We are dynamically generating the runsettings file as it contains the hard-coded path to our custom dotnet test shared framework and some dynamic code coverage settings. I think that approach is fine as we copy it into the app's output directory. I will share our generated runsettings file later.

@ViktorHofer
Copy link
Member

Just as a note, this allows us to use VS and Test Explorer without any env var modifications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants