[Helix] Better logging (flaky tests, logged tests) #18274
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
the
area-infrastructure
HaoK
changed the title
|
Please add @dotnet/aspnet-build as a reviewer when this is ready |
|
@HaoK if you could take a look into where I can put logs that would be picked up by azdo, that'd be useful. If we need to talk with helix folks as well. |
|
@jkotalik yup, I see that the directory is there now: It looks like they are already searching for .log files to automatically upload, so maybe we just need to place them where they are scanning: |
|
Alright, who should we contact in Helix for this issue? |
|
@jkotalik which issue? Helix itself has had a few problems over the last couple of days. But, moving files we want into the expected location seems reasonable. |
|
@dougbu unrelated to outages. We are specifically trying to find a location to put log files where they would be uploaded to azdo and correlated with the test names themselves. |
|
The console logs are associated w/ the tests automatically. If we want to upload more, I believe placing additional files in the same location as those logs is the correct approach. @MattGal is there a variable pointing to the per-test upload location? Anything else we should be aware of? |
Yes put them in $HELIX_WORKITEM_UPLOAD_ROOT / %HELIX_WORKITEM_UPLOAD_ROOT% and read back via the files API. |
|
Is $HELIX_WORKITEM_UPLOAD_ROOT only defined on the helix machine or are they on the azdo machine that builds the runtime? The problem we are running into is we'd define a folder to write logs to at build time on the azdo machine which is then sent to helix. This folder we define would need to be $HELIX_WORKITEM_UPLOAD_ROOT. Otherwise, we may need to hard code it. Also, where is HELIX_WORKITEM_UPLOAD_ROOT defined? I looked in arcade and didn't see anything. |
|
|
I think something like the following should work, we just output the files into |
|
Yay, almost there now, the files show up in the attachments But clicking through to them doesn't result in anything, so maybe they aren't being uploaded properly @MattGal any ideas? Here's the releveant console log output it found the logs, but the links don't go through properly |
|
some of those files have unusual characters that might need escaping but otherwise this is looking buggy, taking a peek. |
|
@HaoK this is very interesting because I can't find any file events for non-console logs for that whole job, and from the harness log it only ever lists one file uploading (the console log). From looking at the console log here there are entries like this: ... which means that if the file is found in /datadisks/disk1/work/B56509B9/w/AB37097F/e/SendMediumData_Success.log, it's NOT in $HELIX_WORKITEM_UPLOAD_ROOT. I believe this is a bug in the Arcade code trying to make this HTML file (which is weird, I thought it was made on the proverbial "other side" of this connection) and will look for that. As a workaround, if you start copying your logs (or just creating them) in |
|
Yeah so my first attempt did that, and the files didn't show up as links, so i tried copying them lower to where the console log is. I can try copying them to both locations and see if that does the trick. |
From my reading of the reporter that's going to result in double links where one is always non-existent, but it's still useful-ish data to try. |
|
🎈 So that did the trick, looks like there's a bug somewhere for you guys, but it takes putting the files in both locations for the links to get generated and be clickable. @jkotalik can you take a quick look at the actual files and make sure that's all of them/what you expect. I just cracked open a few and they look legit to me |
|
Where should I be looking? I don't see it here: |
|
@jkotalik They show up imbedded as links next to the console log, so you have to download the logs.html and they are in that file |
|
Yay! I see the logs as well. |
|
I'll remove the failing assert once this PR is signed off, since without a failing test you can't actually see the logs |
Yes, this is expected because in python |
|
@MattGal uh oh, looks like the xunit-reporter only works for xml files? Not the trx? From the logs: |
Sorry about that, I filed dotnet/arcade#4655 and we'll discuss it next week. |
HaoK
changed the title
|
@HaoK can we revert the test change and merge this soon? Ideally, we can get it merged within the next few hours. |
|
Sure, was waiting for @dotnet/aspnet-build to review first |
dougbu
approved these changes
Jan 21, 2020
eng/helix/content/runtests.sh
Outdated
| @@ -103,10 +103,14 @@ fi | |||
|
|
|||
| FLAKY_FILTER="Flaky:All=true|Flaky:Helix:All=true|Flaky:Helix:Queue:All=true|Flaky:Helix:Queue:$helix_queue_name=true" | |||
| echo "Running known-flaky tests." | |||
| $DOTNET_ROOT/dotnet vstest $test_binary_path --TestCaseFilter:"$FLAKY_FILTER" | |||
| $DOTNET_ROOT/dotnet vstest $test_binary_path --logger:trx --TestCaseFilter:"$FLAKY_FILTER" | |||
There was a problem hiding this comment.
Where do the TRX files end up? I wonder because only *.log files are copied into the magic $HELIX_WORKITEM_UPLOAD_ROOT directory or its parent.
There was a problem hiding this comment.
Trx files are automatically uploaded by the helix infrastructure to azdo, this is how they show up in the test rules tabs, so they come for 'free' to us
There was a problem hiding this comment.
Backing out the trx on flaky run since its causing the jobs to fail
|
Will merge this once PR goes green, I'll need to update the windows version of runtests in a separate PR, I'll add those changes to #17109 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.