Add benchmark for quadratic hash performance

[natecook1000]

This is a hopefully reasonably quick benchmark for the issue described in SR-3268. There's a fix in ca99ee2c342e07a277938ba2bb1b5bfd418ce0c2 that can be tested once the benchmark is in.

(This includes an update to the CMakeList template, since the autogenerated file was edited in a previous commit.)

[lplarson]

@natecook1000 Your CMakeLists.txt_template changes aren't in the actual CMakeLists.txt file. Run the scripts/generate_harness/generate_harness.py script to generate it from the template.

[natecook1000]

The problem was that someone made changes to the generated version and checked that in — my changes to the template only reflect that change.

[lplarson]

@swift-ci benchmark

[swift-ci]

Build comment file:

Optimized (O)

---

[lplarson]

• /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver compare --log-dir /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift --compare-script /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift/benchmark/scripts/compare_perf_tests.py
  Traceback (most recent call last):
  File "/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift/benchmark/scripts/compare_perf_tests.py", line 371, in
  sys.exit(main())
  File "/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift/benchmark/scripts/compare_perf_tests.py", line 154, in main
  ratio = (old_results[key] + 0.001) / (new_results[key] + 0.001)
  KeyError: 'MapReduceAnyCollectionShort'
  Traceback (most recent call last):
  File "/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift/benchmark/scripts/compare_perf_tests.py", line 371, in
  sys.exit(main())
  File "/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift/benchmark/scripts/compare_perf_tests.py", line 154, in main
  ratio = (old_results[key] + 0.001) / (new_results[key] + 0.001)
  KeyError: 'MapReduceAnyCollectionShort'
  branch/branch comparison skipped: no previous branch logs
  Comparing master/Benchmark_O-20170227132347.log HEAD/Benchmark_O-20170227144206.log ...
  Comparing master/Benchmark_Onone-20170227133644.log HEAD/Benchmark_Onone-20170227145919.log ...
  -- check-swift-benchmark-macosx-x86_64 finished --

[natecook1000]

@lplarson I don't think it knows how to calculate the benchmark comparison when there's a new one added; had the same problem with #6382.

[natecook1000]

@swift-ci Please smoke test

[lplarson]

You're right. LGTM.

[lplarson]

@natecook1000 My apologies for missing this. It appears you're trying to compare dict1 and dict2 but you have dict2 on both sides.

[natecook1000]

You're so right! I think that will still defeat inlining, but I'll fix ASAP. Thanks!

[palimondo]

@natecook1000:

This is a hopefully reasonably quick benchmark

On my Late 2008 MBP, it takes 40s to run 1 iteration in O build and 640s in Onone build! For 20 iterations, my machine spends more than 3,5 hours on this single performance test. Could we adjust the size to something more reasonable? It should take less than a second to run one iteration of test run, the benchmark driver will up the N in the outer loop as necessary to run for about a second.

// cc @lplarson @dabrahams

[palimondo]

The CI machine is about 10x faster than my MBP, but it still takes 6s in O, and 50s in Onone build per iteration. That is overall about 20 minutes on this single test.

[natecook1000]

The idea was to actually fix this issue in short order so the test wouldn't actually be quadratic. We definitely don't need to be slowing down the entire benchmarking suite if we don't have a fix in flight.

[palimondo]

Umm... you've referenced fix in commit ca99ee2, did that not land? I'm confused about where that one commit comes from...

[palimondo]

It seems it is in the tree... did that not fix the issue?