[benchmark] Using tags for benchmarks #12242
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
October 2, 2017 17:10
The idea being, we need to decide what benchmarks to run solely based on tags. `--tag` allows to list all tags that are required; `--skip-tags` allows to skip benchmarks that have any of those tags. By default, skip-tags list contains .unstable and .String, which results in the same subset of benchmarks as before.
|
@swift-ci Please smoke test |
|
@swift-ci Please smoke benchmark |
shajrawi
reviewed
Oct 3, 2017
benchmark/utils/main.swift
Outdated
| _ tags: [BenchmarkCategory] | ||
| ) { | ||
| registerBenchmark( | ||
| BenchmarkInfo(name: name, runFunction: function, tags: tags)) |
There was a problem hiding this comment.
@atrick 's point with BenchmarkInfo is to slowly deprecate the 'main' file / migrate towards having the tags inside the benchmark's source file. replacing addTo with this runs counter to that.
There was a problem hiding this comment.
I think this change is fine. I just need to provide new instructions to the one person who was using --registered.
Build comment file:Optimized (O)Regression (3)
Improvement (6)
No Changes (320)
Added (4)
Unoptimized (Onone)Regression (3)
Improvement (3)
No Changes (323)
Added (4)
Hardware Overview |
|
Awesome! Thanks. |
|
@swift-ci Please smoke test |
|
@swift-ci Please smoke benchmark |
|
Hopefully the list of executed benchmarks will be the same ;-) |
|
@swift-ci Please smoke test Linux platform |
|
@swift-ci Please smoke benchmark |
gottesmm
reviewed
Oct 4, 2017
benchmark/utils/TestsUtils.swift
Outdated
| @@ -65,6 +65,9 @@ public enum BenchmarkCategory : String { | |||
| // reimplementing or call into code paths that have known opportunities for | |||
| // significant optimization. | |||
| case cpubench | |||
|
|
|||
| // Explict skip marker | |||
There was a problem hiding this comment.
Darn, I will have to rerun the tests =(
| // | ||
| // --skip-tags=array,set | ||
| // | ||
| // FIXME: If we used Error instead of .fail, then we could have a cleaner |
There was a problem hiding this comment.
I am fine if you want to use Error here. Really we just need a better argument parsing framework. I have one that is partially finished if you want to look at it. Or maybe once we have swiftpm we can just take one from a package.
There was a problem hiding this comment.
I copied that comment from a block above. Did not really pay much attention to what it says. But yeah, a general purpose arg parsing lib would be nice in many places.
| @@ -107,9 +107,12 @@ struct TestConfig { | |||
| /// The filters applied to our test names. | |||
| var filters = [String]() | |||
|
|
|||
| /// The tag that we want to run | |||
| /// The tags that we want to run | |||
There was a problem hiding this comment.
Btw, one other thing that we could do to make this code even better is to use an OptionSet instead of a plain Set. I just forgot to make this change.
There was a problem hiding this comment.
Yeah. but for that the BenchmarkCategory would have to become backed by an Int, and I think there is some reliance on them being Strings.
Build comment file:Optimized (O)Regression (9)
Improvement (13)
No Changes (304)
Added (4)
Removed (3)
Unoptimized (Onone)Regression (4)
Improvement (9)
No Changes (313)
Added (4)
Removed (3)
Hardware Overview |
|
Soo... It looks like we found a benchmark that was never executed =) |
|
No, seriously... |
|
@swift-ci Please clean test Linux platform |
|
Build failed |
|
@swift-ci Please smoke test macOS platform |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing the skip-tags and moving all the benchmarks to using tags to decide which tests to run instead of manually splitting them into separate categories.