[benchmark] Reduce unreasonable setup times #20123
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The DictionaryKeysContains used unreasonably large dictionary to demonstrate the failed O(n) instead of O(1) performance in case of Cocoa Dictionary. The setup of 1M element dictionary took 8 seconds on my old machine! The old pathological behavior can be equaly well demonstrated with a much smaller dictionary. (Validated by modifying `public func _customContainsEquatableElement` in `Dictionary.swift` to `return _variant.index(forKey: element) != nil`) The reported performance with correct O(1) behavior is unchanged.
Reduced the time to run the setUpFunction from 2.2s to 380ms on my ancient computer… This should fit well under 200ms on more modern machines.
|
@eeckstein Please review 🙏 |
|
@swift-ci benchmark |
Build comment file:Code size: -O
Performance: -Osize
Code size: -Osize
How to read the dataThe tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR. Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein). Hardware Overview |
|
@swift-ci smoke test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reduces setup time of 3 benchmarks that took unreasonably long to prepare their workloads. It brings them in compliance with the new
BenchmarkDoctorrule: setUpFunction should take reasonable time (no more than 200ms). More details in the individual commit descriptions.