[windows][test] Ensure the ordering of stdout and stderr messages.

It is possible that the rest of the platforms are relying in some not
clearly documented behaviour that the stdout is flushed before stderr
can be used, or something similar. Windows didn't seem to like that, and
was sometimes outputting the stderr messages interlaced with the stdout
messages (specially in the Azure testing for VS2017). The initial
solution was adding -DAG to some CHECK lines, but that doesn't cover all
the possibilities, and wasn't enabled for all the checks.

I tried to fix it in several ways, but none of them were perfect, and
many of them were deadlocking. There's a fundamental difference between
others and Windows and that is that for others stdout seems to have a
little bit of an edge in being treated first. I tried to get the Windows
code closer to that idea, but I have no luck. Some of the approaches
were using the main thread to read from the threads reading stdout and
stderr; using only the main thread and IOCP on the pipes reading from
the child process; and flushing after every output to stdout. None of
them were perfect.

The final solution is a hack, but it seems to not fail when I run the
test repeatedly in my machine, while other approaches were failing at
least once before I discarded them. The solution is including a small
sleep (1 millisecond) in the Windows code. This should yield the
execution time slice to other thread/process, which seems to do the
trick and keep the stdout before the stderr.

Hopefully this fixes the errors in Azure, and doesn't affect the rest of
the testing machines. Being a really small sleep also should not affect
the duration of the test itself.

Author: drodriguez

validation-test/StdlibUnittest/CrashingTests.swift

@@ -5,10 +5,30 @@
 
 import StdlibUnittest
 
+#if os(Windows)
+// HACK: It seems that other platforms might be lucky and the stdout and stderr
+// are being sent to the parent process in the order they are used. However, in
+// Windows the result of a print followed by a fatalError is not always ordered
+// the same in the parent. To avoid a random order, we add Sleep(1) before the
+// fatalError calls, which yields enough time to other threads so the output is
+// ordered like in other platforms.
+  import WinSDK
+#endif
+
 
 _setOverrideOSVersion(.osx(major: 10, minor: 9, bugFix: 3))
 _setTestSuiteFailedCallback() { print("abort()") }
 
+private func fatalErrorWithDelayIfNeeded(
+  _ message: @autoclosure () -> String = String(),
+  file: StaticString = #file, line: UInt = #line
+) -> Never {
+  #if os(Windows)
+    Sleep(1)
+  #endif
+  fatalError(message, file: file, line: line)
+}
+
 //
 // Test that harness aborts when a test crashes during a test run.
 //
@@ -17,10 +37,10 @@ var TestSuiteCrashes = TestSuite("TestSuiteCrashes")
 
 TestSuiteCrashes.test("crashesUnexpectedly1") {
   print("crashesUnexpectedly1")
-  fatalError("This should crash")
+  fatalErrorWithDelayIfNeeded("This should crash")
 }
-// CHECK-DAG: stdout>>> crashesUnexpectedly1
-// CHECK-DAG: stderr>>> Fatal error: This should crash:
+// CHECK: stdout>>> crashesUnexpectedly1
+// CHECK: stderr>>> Fatal error: This should crash:
 // CHECK: stderr>>> CRASHED: SIG
 // CHECK: [     FAIL ] TestSuiteCrashes.crashesUnexpectedly1
 
@@ -41,7 +61,7 @@ TestSuiteCrashes.test("fails1") {
 
 TestSuiteCrashes.test("crashesUnexpectedly2") {
   print("crashesUnexpectedly2")
-  fatalError("This should crash")
+  fatalErrorWithDelayIfNeeded("This should crash")
 }
 // CHECK: stdout>>> crashesUnexpectedly2
 // CHECK: stderr>>> Fatal error: This should crash:
@@ -66,7 +86,7 @@ TestSuiteCrashes.test("fails2") {
 TestSuiteCrashes.test("crashesAsExpected1") {
   print("crashesAsExpected1")
   expectCrashLater()
-  fatalError("This should crash")
+  fatalErrorWithDelayIfNeeded("This should crash")
 }
 // CHECK: stdout>>> crashesAsExpected1
 // CHECK: stderr>>> Fatal error: This should crash:
@@ -91,7 +111,7 @@ TestSuiteCrashes.test("fails3") {
 TestSuiteCrashes.test("crashesUnexpectedlyXfail")
   .xfail(.osxBugFix(10, 9, 3, reason: "")).code {
   print("crashesUnexpectedlyXfail")
-  fatalError("This should crash")
+  fatalErrorWithDelayIfNeeded("This should crash")
 }
 // CHECK: stdout>>> crashesUnexpectedlyXfail
 // CHECK: stderr>>> Fatal error: This should crash:
@@ -102,7 +122,7 @@ TestSuiteCrashes.test("crashesAsExpectedXfail")
   .xfail(.osxBugFix(10, 9, 3, reason: "")).code {
   print("crashesAsExpectedXfail")
   expectCrashLater()
-  fatalError("This should crash")
+  fatalErrorWithDelayIfNeeded("This should crash")
 }
 // CHECK: stdout>>> crashesAsExpectedXfail
 // CHECK: stderr>>> Fatal error: This should crash:
@@ -113,7 +133,7 @@ TestSuiteCrashes.test("crashesWithMessagePasses")
   .crashOutputMatches("This should crash").code {
   print("abcd")
   expectCrashLater()
-  fatalError("This should crash")
+  fatalErrorWithDelayIfNeeded("This should crash")
 }
 // CHECK: stdout>>> abcd
 // CHECK: stderr>>> Fatal error: This should crash:
@@ -124,7 +144,7 @@ TestSuiteCrashes.test("crashesWithMessageFails")
   .crashOutputMatches("This should crash").code {
   print("This should crash")
   expectCrashLater()
-  fatalError("unexpected message")
+  fatalErrorWithDelayIfNeeded("unexpected message")
 }
 // CHECK: stdout>>> This should crash
 // CHECK: stderr>>> Fatal error: unexpected message:
@@ -139,7 +159,7 @@ TestSuiteCrashes.test("crashesWithMultipleMessagesPasses")
   .code {
   print("abcd")
   expectCrashLater()
-  fatalError("This should crash and your little dog too")
+  fatalErrorWithDelayIfNeeded("This should crash and your little dog too")
 }
 // CHECK: stdout>>> abcd
 // CHECK: stderr>>> Fatal error: This should crash and your little dog too:
@@ -154,7 +174,7 @@ TestSuiteCrashes.test("crashesWithMultipleMessagesFails")
 .code {
   print("This should crash")
   expectCrashLater()
-  fatalError("unexpected message and your little dog too")
+  fatalErrorWithDelayIfNeeded("unexpected message and your little dog too")
 }
 // CHECK: stdout>>> This should crash
 // CHECK: stderr>>> Fatal error: unexpected message and your little dog too: