Flake Finder

[analogrelay]

A PR with no changes to help tease out flakiness in our PR builds.

cc @pakrym @dougbu @ryanbrandenburg

[jkotalik]

Make more of these 😄

[analogrelay]

Make more of these 😄

Don't want to overwhelm the queue, but I'll add some more... slowly :)

[analogrelay]

Flakiness found! #8100

[HaoK]

Longer term, maybe what we should do is to just put them in their own little 'troll island' and run all the flaky tests together so they at least run, the upside of this is we can see just how flaky they are, and its also easier for people to look into them if they are still running (maybe we just have a permanent flaky-test-hell-branch that has an additional check that runs all the tests including the flaky ones so it doesn't make master PRs red)

[dougbu]

Hmm… For example, could create a new xUnit.Net attribute that skips tests except when some environment variable is set to 1, true, True et cetera. A non-required (Helix?) job in every build could set the environment variable and otherwise do what it normally does.

[HaoK]

Yeah we just need an additional [FlakyTest] badge of shame and can have it do the right things

[analogrelay]

I like it. It helps alleviate some of the concerns around skipping flaky tests immediately (rather than having them be investigated first).

[analogrelay]

/azp run AspNetCore-helix-test