[test] Run all the benchmarks once as a validation test

[jrose-apple]

This is most useful in +Asserts builds of the stdlib, which isn't a useful configuration for actual benchmarking, but could still uncover issues in the implementation. (There actually is one right now found by SuperChars, SR-9203, so I can't land this yet.)

[eeckstein]

nice!

[palimondo]

This makes perfect sense.

BTW, we have been discussing with @eeckstein and @shahmishal building the benchmarks as part of smoke test in #21477 (comment).

Running with the parameters you have here currently takes 7 seconds on my 2008 MBP and when I'm done with all the Janitor Duty: Legacy Factor cleanup, this should be sub-second... so then we should probably move this to test, instead of validation-test.

[palimondo]

@swift-ci please test

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 755b290

[palimondo]

An this is why needed this yesterday ;-)

Crash Details:

705,SumUsingReduce,1,1265,1265,1265,0,1265
706,SumUsingReduceInto,1,1723,1723,1723,0,1723

--
Command Output (stderr):
--
Fatal error: file /Users/buildnode/jenkins/workspace/swift-PR-osx/branch-master/swift/stdlib/public/core/Character.swift, line 82
Current stack trace:
0    libswiftCore.dylib                 0x0000000102923e60 _swift_stdlib_reportFatalErrorInFile + 116
1    libswiftCore.dylib                 0x000000010283d450 specialized closure #1 in closure #1 in closure #1 in _fatalErrorMessage(_:_:file:line:flags:) + 306
2    libswiftCore.dylib                 0x00000001027f7530 specialized closure #1 in closure #1 in _fatalErrorMessage(_:_:file:line:flags:) + 113
3    libswiftCore.dylib                 0x00000001027f78f0 specialized closure #1 in _fatalErrorMessage(_:_:file:line:flags:) + 99
4    libswiftCore.dylib                 0x00000001027f7c90 specialized _fatalErrorMessage(_:_:file:line:flags:) + 561
5    libswiftCore.dylib                 0x0000000102809460 specialized Character._invariantCheck() + 179
6    libswiftCore.dylib                 0x00000001025bc950 Character._invariantCheck() + 9
7    Benchmark_O                        0x00000001023fe560 globalinit_33_BF26B45B13997F01E53CA304BE2C1E8C_func1 + 2840
8    libdispatch.dylib                  0x00007fff6d1f4db0 _dispatch_client_callout + 8
9    libdispatch.dylib                  0x00007fff6d1f4d42 dispatch_once_f + 41
10   libswiftCore.dylib                 0x00000001028e0600 swift_once + 25
11   Benchmark_O                        0x00000001023fe4a0 closure #1 in variable initialization expression of SuperChars + 40
12   Benchmark_O                        0x000000010212d310 TestRunner.run(_:) + 1578
13   Benchmark_O                        0x0000000102130dd0 main() + 3694
14   Benchmark_O                        0x000000010242d000 main + 17102
15   libdyld.dylib                      0x00007fff6d22e014 start + 1
/Users/buildnode/jenkins/workspace/swift-PR-osx/branch-master/buildbot_incremental/swift-macosx-x86_64/validation-test-macosx-x86_64/benchmark/Output/run-benchmarks.test-sh.script: line 1:  2410 Illegal instruction: 4  /Users/buildnode/jenkins/workspace/swift-PR-osx/branch-master/buildbot_incremental/swift-macosx-x86_64/bin/Benchmark_O --num-iters=1 --num-samples=1

Edit: Oh… but that is the SuperChars crash. Fix pending: #21761. Move along…

[palimondo]

@swift-ci please test

[jrose-apple]

I put it in validation-test for two reasons:

1. It's really unlikely to break in practice. Benchmarks aren't usually exercising the stdlib in unusual ways.
2. I tend to not rebuild the benchmark drivers when doing local development, but that means lit thinks I've built them even though they're out of date. And I usually only run the test/ tests locally, since I know PR testing will catch the rest. I guess I'm already ignoring the existing failures here, though.

So, yeah, maybe it does make sense to move it to test/. I'll merge it as is for now, though, so we can get the coverage immediately.