Skip to content

benchmark: add naive string finding #26744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 30, 2019
Merged

benchmark: add naive string finding #26744

merged 1 commit into from
Aug 30, 2019

Conversation

weissi
Copy link
Contributor

@weissi weissi commented Aug 20, 2019

FindStringNaive is a simple benchmark which implements a naive String
finding algorithm that currently shows a lot of ARC traffic, hopefully
to be reduced in the future.

this is a benchmark for https://bugs.swift.org/browse/SR-11335

@weissi weissi requested a review from gottesmm August 20, 2019 17:32
@weissi weissi force-pushed the jw-bench-find-string branch from 45acc7a to 7f5a834 Compare August 20, 2019 17:59
@swiftlang swiftlang deleted a comment from swift-ci Aug 20, 2019
@weissi weissi force-pushed the jw-bench-find-string branch from 6d8b95f to 7964013 Compare August 21, 2019 07:20
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@weissi weissi force-pushed the jw-bench-find-string branch from 7964013 to ad3b3ad Compare August 21, 2019 09:44
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@weissi weissi force-pushed the jw-bench-find-string branch 2 times, most recently from 802afee to 08da14c Compare August 21, 2019 10:35
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@weissi weissi force-pushed the jw-bench-find-string branch from 08da14c to 8dbd969 Compare August 21, 2019 11:26
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@weissi weissi force-pushed the jw-bench-find-string branch from 8dbd969 to 84a025e Compare August 21, 2019 14:07
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@weissi weissi force-pushed the jw-bench-find-string branch from 84a025e to db64d7c Compare August 21, 2019 14:32
@swiftlang swiftlang deleted a comment from swift-ci Aug 21, 2019
@weissi
Copy link
Contributor Author

weissi commented Aug 21, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

Regression OLD NEW DELTA RATIO
CharIndexing_punctuated_unicodeScalars_Backwards 720 800 +11.1% 0.90x (?)
FlattenListFlatMap 3876 4239 +9.4% 0.91x (?)
MapReduceSequence 397 428 +7.8% 0.93x (?)
 
Improvement OLD NEW DELTA RATIO
ObjectiveCBridgeStubNSDateRefAccess 196 174 -11.2% 1.13x (?)
 
Added MIN MAX MEAN MAX_RSS
FindStringNaive.ArrayUTF8 436 444 439 —
FindStringNaive.String 316 316 316 —
FindStringNaive.Substring 637 637 637 —

Code size: -O

Performance: -Osize

Regression OLD NEW DELTA RATIO
ObjectiveCBridgeStubDateAccess 130 152 +16.9% 0.86x
FlattenListLoop 2168 2379 +9.7% 0.91x (?)
 
Improvement OLD NEW DELTA RATIO
Dictionary4 205 164 -20.0% 1.25x
CharacterLiteralsLarge 74 67 -9.5% 1.10x
StringHasPrefixAscii 1280 1170 -8.6% 1.09x (?)
 
Added MIN MAX MEAN MAX_RSS
FindStringNaive.ArrayUTF8 437 444 439 —
FindStringNaive.String 315 315 315 —
FindStringNaive.Substring 637 649 641 —

Code size: -Osize

Performance: -Onone

Improvement OLD NEW DELTA RATIO
ObjectiveCBridgeStubFromNSStringRef 216 110 -49.1% 1.96x (?)
Breadcrumbs.MutatedUTF16ToIdx.ASCII 15 14 -6.7% 1.07x (?)
 
Added MIN MAX MEAN MAX_RSS
FindStringNaive.ArrayUTF8 188100 188277 188197 —
FindStringNaive.String 600 601 601 —
FindStringNaive.Substring 760 760 760 —

Code size: -swiftlibs

✅ Benchmark Check Report
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

@weissi
Copy link
Contributor Author

weissi commented Aug 21, 2019

@gottesmm the benchmark is now happy and gives me a green tick. How does this look to you?

@weissi
Copy link
Contributor Author

weissi commented Aug 21, 2019

@milseman also turns out that finding a UInt8 in a [UInt8] is 100x(!!) faster than finding a Character in a String (despite it being UTF-8 with ascii only chars).

@gottesmm
Copy link
Contributor

@weissi Just squash the commits then LGTM.

@Catfish-Man
Copy link
Contributor

Catfish-Man commented Aug 21, 2019 •
edited
Loading

@milseman also turns out that finding a UInt8 in a [UInt8] is 100x(!!) faster than finding a Character in a String (despite it being UTF-8 with ascii only chars).

This is most likely because of having to treat CRLF as a single Character. It prevents a lot of the obvious fast paths.

@weissi weissi force-pushed the jw-bench-find-string branch from db64d7c to 7f9756e Compare August 21, 2019 18:17
@weissi
Copy link
Contributor Author

weissi commented Aug 21, 2019

@gottesmm done, thank you!

@Catfish-Man
Copy link
Contributor

Please don't edit my comments (??)

@weissi
Copy link
Contributor Author

weissi commented Aug 22, 2019

Please don't edit my comments (??)

So sorry!! I was on my phone as must have clicked 'edit' instead of 'quote reply' and didn't notice :'(

milseman
milseman reviewed Aug 22, 2019
View reviewed changes
@weissi weissi force-pushed the jw-bench-find-string branch from 7f9756e to 43a1517 Compare August 28, 2019 16:02
@weissi
Copy link
Contributor Author

weissi commented Aug 28, 2019

@swift-ci please benchmark

@weissi weissi force-pushed the jw-bench-find-string branch from 43a1517 to 8726280 Compare August 28, 2019 16:52
@swiftlang swiftlang deleted a comment from swift-ci Aug 28, 2019
@weissi
Copy link
Contributor Author

weissi commented Aug 28, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Build failed before running benchmark.

@weissi weissi force-pushed the jw-bench-find-string branch from 8726280 to 6e5e6c1 Compare August 28, 2019 17:32
@weissi
Copy link
Contributor Author

weissi commented Aug 28, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Build failed before running benchmark.

@weissi
benchmark: add naive string finding
f758d22 
FindStringNaive is a simple benchmark which implements a naive String
finding algorithm that currently shows a lot of ARC traffic, hopefully
to be reduced in the future.
@weissi weissi force-pushed the jw-bench-find-string branch from 6e5e6c1 to f758d22 Compare August 28, 2019 18:57
@weissi
Copy link
Contributor Author

weissi commented Aug 28, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

Improvement OLD NEW DELTA RATIO
CharacterLiteralsLarge 108 97 -10.2% 1.11x (?)
CharacterLiteralsSmall 340 317 -6.8% 1.07x (?)
 
Added MIN MAX MEAN MAX_RSS
FindString.Loop1.Array 571 572 571 —
FindString.Loop1.Substring 803 803 803 —
FindString.Rec3.Array 237 238 237 —
FindString.Rec3.String 206 208 207 —
FindString.Rec3.Substring 244 246 245 —

Code size: -O

Performance: -Osize

Regression OLD NEW DELTA RATIO
FlattenListLoop 5036 5644 +12.1% 0.89x (?)
ObjectiveCBridgeStubDateMutation 257 285 +10.9% 0.90x
Array2D 7520 8112 +7.9% 0.93x
RandomShuffleLCG2 816 880 +7.8% 0.93x (?)
 
Improvement OLD NEW DELTA RATIO
ObjectiveCBridgeStubFromNSDateRef 4200 3720 -11.4% 1.13x (?)
ObjectiveCBridgeStubFromNSStringRef 176 156 -11.4% 1.13x (?)
ObjectiveCBridgeStubDateAccess 257 228 -11.3% 1.13x
NSStringConversion.UTF8 834 770 -7.7% 1.08x (?)
 
Added MIN MAX MEAN MAX_RSS
FindString.Loop1.Array 571 572 571 —
FindString.Loop1.Substring 803 803 803 —
FindString.Rec3.Array 237 237 237 —
FindString.Rec3.String 211 213 212 —
FindString.Rec3.Substring 243 245 244 —

Code size: -Osize

Performance: -Onone

Added MIN MAX MEAN MAX_RSS
FindString.Loop1.Array 266727 295816 276477 —
FindString.Loop1.Substring 989 990 989 —
FindString.Rec3.Array 14355 14498 14415 —
FindString.Rec3.String 935 935 935 —
FindString.Rec3.Substring 897 898 898 —

Code size: -swiftlibs

✅ Benchmark Check Report
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

milseman
milseman approved these changes Aug 29, 2019
View reviewed changes
Copy link
Member

@milseman milseman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@weissi
Copy link
Contributor Author

weissi commented Aug 29, 2019

@swift-ci please test

@swift-ci
Copy link
Contributor

Build failed
Swift Test Linux Platform
Git Sha - 43a151738c2abfad8548c7360b0cc97522e7c0ac

@swift-ci
Copy link
Contributor

Build failed
Swift Test OS X Platform
Git Sha - 43a151738c2abfad8548c7360b0cc97522e7c0ac

@weissi
Copy link
Contributor Author

weissi commented Aug 29, 2019

@swift-ci smoke test and merge

@gottesmm gottesmm merged commit d128664 into swiftlang:master Aug 30, 2019
@weissi weissi deleted the jw-bench-find-string branch August 30, 2019 11:15
palimondo
palimondo reviewed Sep 3, 2019
View reviewed changes
benchmark/single-source/FindStringNaive.swift
@inline(never)
func runBenchRecursive3ArrayOfUTF8(iterations: Int) {
for _ in 0 ..< iterations {
precondition(findThreeOnUTF8Collection(longArrayFoFoFoFox!,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eeckstein @gottesmm Is the use of precondition here fully equivalent to traditional CheckResults we use in all other benchmarks?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, CheckResult is better. It should be slightly less overhead because it's inlinable.

@weissi weissi mentioned this pull request Aug 20, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@palimondo palimondo palimondo left review comments

@eeckstein eeckstein eeckstein left review comments

@milseman milseman milseman approved these changes

@gottesmm gottesmm gottesmm approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@weissi @swift-ci @gottesmm @Catfish-Man @milseman @palimondo @eeckstein