[benchmark] Add benchmark for [AnyHashable: Any] with String keys

[lorentey]

We don't have adequate benchmark coverage for AnyHashable. This PR adds a couple of benchmarks for the most important usecase, [AnyHashable: Any] with String keys. (An undecorated NSDictionary gets imported as such into Swift.)

[lorentey]

@swift-ci please smoke test

[lorentey]

@swift-ci please benchmark

[swift-ci]

Build comment file:

Performance: -O

TEST	MIN	MAX	MEAN	MAX_RSS
Added
DictionaryOfAnyHashableStrings_insert	5207	5553	5326	—
DictionaryOfAnyHashableStrings_lookup	3898	3955	3919	—

Performance: -Osize

TEST	MIN	MAX	MEAN	MAX_RSS
Added
DictionaryOfAnyHashableStrings_insert	6804	7104	6905	—
DictionaryOfAnyHashableStrings_lookup	4421	4526	4457	—

Performance: -Onone

TEST	MIN	MAX	MEAN	MAX_RSS
Added
DictionaryOfAnyHashableStrings_insert	7579	7782	7657	—
DictionaryOfAnyHashableStrings_lookup	4687	4730	4705	—

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

---

[lorentey]

FAIL: test_dsym (lldbsuite.test.lldbtest.TestSwiftPrivateVarReplInC)
FAIL: test_dwarf (lldbsuite.test.lldbtest.TestSwiftPrivateVarReplInC)

Hm; these failures are unrelated.

[lorentey]

@swift-ci test

[adrian-prantl]

@swift-ci test

[gottesmm]

@swift-ci smoke test os x platform

[lorentey]

@swift-ci smoke test linux platform

[palimondo]

@lorentey Since these are new benchmarks, would you mind if I lowered the base workload on them by a factor of 10 and used the CamelCase naming convention, when I make this change? (DictionaryOfAnyHashableStringsInsert, DictionaryOfAnyHashableStringsLookup)

[lorentey]

Feel free to change the iteration multiplier!

Please don't change the names, though -- the underscore visually separates the data structure from the operation, and merging the two components together will make it harder for me to find the particular result I'm interested in. Is there a technical reason to require camel case?

[palimondo]

No technical reason. Just adherence to Swift Naming Guidelines.

[lorentey]

Benchmark names aren't APIs.

If anything, I'd actually prefer to name microbenchmarks directly after the member signature they test. E.g., I'd name the lookup test here something like [AnyHashable: Any].subscript(String) getter. But this may be a minority opinion. :)

[palimondo]

Well, most of the SBS currently consists of composite benchmarks. Individual method tests are in absolute minority. IMO the driving factor for naming should be for them to fit into reports in GitHub without causing the table to scroll horizontally ;-) Going from what's currently in there, UpperCamelCase is the prevailing convention, with absolute minority with one underscore mixed in… The rule I've put in the BenchmarkDoctor for now recommends names up to 40 characters and the use of UpperCamelCase from naming guidelines for consistency. Maybe we could go with a dot for individual method tests? I.e. DictionaryOfAnyHashableStrings.insert, DictionaryOfAnyHashableStrings.lookup…
Initially I thought that since benchmark names are just strings, we could go the full way for benchmarks that are this specific, as you suggested [AnyHashable: Any].subscript(String), but the names have to be work as parameters on the command line, so spaces and other symbols that can be interpreted by shell are not usable...
Would you like to discuss this in the forum?

[lorentey]

To me, any separator would be much preferable to DictionaryOfAnyHashableStringsInsert. DictionaryOfAnyHashableStrings is already quite bad, but appending Insert to it makes it unreadable.

Benchmark names are very similar to test names -- some of them are hierarchical and we should have a rule that accommodates that. The dot would work very well!

Names like SetSymmetricDifferenceInt50 didn't bother me as much because their components are more straightforward, but Set<Int>.symmetricDifference() 50% would still read a lot better in any table. However, Set.Int.symmetricDifference.50 is a reasonable compromise between the two, and it's still quite usable.

I think shell limitations aren't really relevant -- it's not too difficult to quote names, especially if we stay away from using apostrophes.

[palimondo]

@lorentey I'm working on cleaning this up. I've lowered the workload to 50 keys. But there is also about 50% setup overhead (run Benchmark_Driver check 181 182 to see for yourself), but I can't find what is causing it. I've been stumped by this for past few hours…

Can you please help me identify the source of setup overhead in DictionaryOfAnyHashableStrings_insert and _update?

[palimondo]

@lorentey ping ^^

[lorentey]

I don't see anything obvious, either. Does it disappear if you do a couple iterations of the benchmark loop in the setup function?

[palimondo]

Nope. I did up to 5x blackHole warmups in the setup, not helping...

[palimondo]

Hah! Finally cracked the mystical setup overhead: one extra iteration. The loop should be either 1...n or 0..<n. I'll fix this in a next Janitor Duty legacyFactor pass.