Skip to content

benchmarks: more work to make benchmarks more stable #18363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jul 31, 2018
Merged

benchmarks: more work to make benchmarks more stable #18363

merged 4 commits into from
Jul 31, 2018

Conversation

eeckstein
Copy link
Contributor

for details see commit list

@eeckstein
Copy link
Contributor Author

@swift-ci smoke benchmark

@eeckstein
Copy link
Contributor Author

@swift-ci smoke test

@swift-ci
Copy link
Contributor

Build comment file:

Optimized (O)

Regression (12)
TEST OLD NEW DELTA SPEEDUP
DictionarySwapAt 7612 10752 +41.3% 0.71x
BinaryFloatingPointPropertiesBinade 25 31 +24.0% 0.81x
AnyHashableWithAClass 91962 109892 +19.5% 0.84x
UTF8Decode_InitFromBytes 1165 1384 +18.8% 0.84x
DictionaryBridgeToObjC_Access 922 1073 +16.4% 0.86x (?)
IterateData 1532 1721 +12.3% 0.89x
StaticArray 9 10 +11.1% 0.90x
ObjectiveCBridgeToNSSet 17792 19221 +8.0% 0.93x (?)
DataAppendArray 5517 5957 +8.0% 0.93x (?)
UTF8Decode_InitFromData_ascii 883 942 +6.7% 0.94x (?)
NopDeinit 30162 32156 +6.6% 0.94x
DictionaryRemove 4371 4634 +6.0% 0.94x
Improvement (15)
TEST OLD NEW DELTA SPEEDUP
PointerArithmetics 34385 31507 -8.4% 1.09x
UTF8Decode_InitFromBytes_ascii 588 539 -8.3% 1.09x (?)
MapReduceLazyCollectionShort 37 34 -8.1% 1.09x
DataCount 37 34 -8.1% 1.09x
DataReplaceSmallBuffer 9580 8816 -8.0% 1.09x (?)
DropLastSequenceLazy 690 640 -7.2% 1.08x
DropLastSequence 691 642 -7.1% 1.08x
Hanoi 2307 2155 -6.6% 1.07x
MapReduceString 50 47 -6.0% 1.06x
DataAppendSequence 22660 21322 -5.9% 1.06x
RandomShuffleDef2 2740 2583 -5.7% 1.06x (?)
DropLastAnyCollectionLazy 23024 21713 -5.7% 1.06x
BinaryFloatingPointPropertiesUlp 37 35 -5.4% 1.06x
CharIteration_utf16_unicodeScalars_Backwards 18744 17787 -5.1% 1.05x
RangeAssignment 353 336 -4.8% 1.05x
No Changes (413)
TEST OLD NEW DELTA SPEEDUP
AngryPhonebook 3774 3812 +1.0% 0.99x (?)
Array2D 2718 2718 +0.0% 1.00x
ArrayAppend 1081 1077 -0.4% 1.00x (?)
ArrayAppendArrayOfInt 800 806 +0.7% 0.99x (?)
ArrayAppendAscii 3907 3895 -0.3% 1.00x (?)
ArrayAppendAsciiSubstring 25354 25073 -1.1% 1.01x
ArrayAppendFromGeneric 809 800 -1.1% 1.01x
ArrayAppendGenericStructs 1430 1412 -1.3% 1.01x (?)
ArrayAppendLatin1 42487 42387 -0.2% 1.00x (?)
ArrayAppendLatin1Substring 138452 138936 +0.3% 1.00x
ArrayAppendLazyMap 1342 1341 -0.1% 1.00x (?)
ArrayAppendOptionals 1433 1431 -0.1% 1.00x (?)
ArrayAppendRepeatCol 1348 1340 -0.6% 1.01x
ArrayAppendReserved 806 805 -0.1% 1.00x (?)
ArrayAppendSequence 1127 1123 -0.4% 1.00x
ArrayAppendStrings 6329 6319 -0.2% 1.00x (?)
ArrayAppendToFromGeneric 800 799 -0.1% 1.00x (?)
ArrayAppendToGeneric 801 801 +0.0% 1.00x
ArrayAppendUTF16 42174 41873 -0.7% 1.01x
ArrayAppendUTF16Substring 136757 137460 +0.5% 0.99x (?)
ArrayInClass 86 85 -1.2% 1.01x
ArrayLiteral 0 0 +0.0% 1.00x
ArrayOfGenericPOD2 151 152 +0.7% 0.99x (?)
ArrayOfGenericRef 4373 4390 +0.4% 1.00x (?)
ArrayOfPOD 186 186 +0.0% 1.00x
ArrayOfRef 4344 4355 +0.3% 1.00x (?)
ArrayPlusEqualArrayOfInt 803 804 +0.1% 1.00x (?)
ArrayPlusEqualFiveElementCollection 5487 5466 -0.4% 1.00x
ArrayPlusEqualSingleElementCollection 1079 1075 -0.4% 1.00x (?)
ArrayPlusEqualThreeElements 1665 1667 +0.1% 1.00x (?)
ArraySubscript 1576 1565 -0.7% 1.01x (?)
ArrayValueProp 8 8 +0.0% 1.00x
ArrayValueProp2 8 8 +0.0% 1.00x
ArrayValueProp3 8 8 +0.0% 1.00x
ArrayValueProp4 8 8 +0.0% 1.00x
BinaryFloatingPointPropertiesNextUp 28 28 +0.0% 1.00x
BitCount 202 211 +4.5% 0.96x
ByteSwap 105 107 +1.9% 0.98x
COWArrayGuaranteedParameterOverhead 10094 10134 +0.4% 1.00x (?)
COWTree 3657 3679 +0.6% 0.99x (?)
CSVParsing2 1619 1621 +0.1% 1.00x (?)
CSVParsingAlt2 1779 1783 +0.2% 1.00x (?)
CSVParsingAltIndices2 786 775 -1.4% 1.01x (?)
CStringLongAscii 3954 3954 +0.0% 1.00x
CStringLongNonAscii 2298 2299 +0.0% 1.00x (?)
CStringShortAscii 3333 3336 +0.1% 1.00x (?)
Calculator 207 208 +0.5% 1.00x
CaptureProp 4099 4092 -0.2% 1.00x (?)
ChainedFilterMap 1248 1246 -0.2% 1.00x (?)
CharIndexing_ascii_unicodeScalars 17395 17452 +0.3% 1.00x
CharIndexing_ascii_unicodeScalars_Backwards 16054 16044 -0.1% 1.00x (?)
CharIndexing_chinese_unicodeScalars 13201 13218 +0.1% 1.00x (?)
CharIndexing_chinese_unicodeScalars_Backwards 12198 12176 -0.2% 1.00x (?)
CharIndexing_japanese_unicodeScalars 20848 20897 +0.2% 1.00x (?)
CharIndexing_japanese_unicodeScalars_Backwards 19160 19167 +0.0% 1.00x (?)
CharIndexing_korean_unicodeScalars 16866 16890 +0.1% 1.00x
CharIndexing_korean_unicodeScalars_Backwards 15598 15509 -0.6% 1.01x
CharIndexing_punctuatedJapanese_unicodeScalars 3134 3139 +0.2% 1.00x
CharIndexing_punctuatedJapanese_unicodeScalars_Backwards 2915 2918 +0.1% 1.00x (?)
CharIndexing_punctuated_unicodeScalars 3927 3935 +0.2% 1.00x (?)
CharIndexing_punctuated_unicodeScalars_Backwards 3653 3666 +0.4% 1.00x (?)
CharIndexing_russian_unicodeScalars 14464 14525 +0.4% 1.00x (?)
CharIndexing_russian_unicodeScalars_Backwards 13347 13339 -0.1% 1.00x (?)
CharIndexing_tweet_unicodeScalars 33973 34128 +0.5% 1.00x
CharIndexing_tweet_unicodeScalars_Backwards 31942 31910 -0.1% 1.00x
CharIndexing_utf16_unicodeScalars 23048 23152 +0.5% 1.00x
CharIndexing_utf16_unicodeScalars_Backwards 23623 23680 +0.2% 1.00x
CharIteration_ascii_unicodeScalars 21636 21447 -0.9% 1.01x
CharIteration_ascii_unicodeScalars_Backwards 14771 14759 -0.1% 1.00x (?)
CharIteration_chinese_unicodeScalars 16363 16226 -0.8% 1.01x
CharIteration_chinese_unicodeScalars_Backwards 11198 11195 -0.0% 1.00x (?)
CharIteration_japanese_unicodeScalars 25920 25697 -0.9% 1.01x
CharIteration_japanese_unicodeScalars_Backwards 17669 17665 -0.0% 1.00x (?)
CharIteration_korean_unicodeScalars 20974 20820 -0.7% 1.01x
CharIteration_korean_unicodeScalars_Backwards 14336 14325 -0.1% 1.00x (?)
CharIteration_punctuatedJapanese_unicodeScalars 3878 3851 -0.7% 1.01x
CharIteration_punctuatedJapanese_unicodeScalars_Backwards 2713 2704 -0.3% 1.00x
CharIteration_punctuated_unicodeScalars 4861 4824 -0.8% 1.01x
CharIteration_punctuated_unicodeScalars_Backwards 3380 3376 -0.1% 1.00x
CharIteration_russian_unicodeScalars 18023 17869 -0.9% 1.01x
CharIteration_russian_unicodeScalars_Backwards 12330 12315 -0.1% 1.00x (?)
CharIteration_tweet_unicodeScalars 43192 42328 -2.0% 1.02x
CharIteration_tweet_unicodeScalars_Backwards 29147 29144 -0.0% 1.00x (?)
CharIteration_utf16_unicodeScalars 28155 27954 -0.7% 1.01x
CharacterLiteralsLarge 5894 5860 -0.6% 1.01x (?)
CharacterLiteralsSmall 217 217 +0.0% 1.00x
CharacterPropertiesFetch 4619 4648 +0.6% 0.99x (?)
CharacterPropertiesPrecomputed 1020 1021 +0.1% 1.00x
CharacterPropertiesStashed 1796 1810 +0.8% 0.99x (?)
CharacterPropertiesStashedMemo 1575 1567 -0.5% 1.01x (?)
ClassArrayGetter2 124 125 +0.8% 0.99x
Combos 497 497 +0.0% 1.00x
DataAccessBytes 1146 1143 -0.3% 1.00x (?)
DataAppendBytes 5407 5321 -1.6% 1.02x (?)
DataAppendDataLargeToLarge 68820 66492 -3.4% 1.04x (?)
DataAppendDataLargeToMedium 35237 36405 +3.3% 0.97x
DataAppendDataLargeToSmall 34574 34550 -0.1% 1.00x (?)
DataAppendDataMediumToLarge 37887 37789 -0.3% 1.00x (?)
DataAppendDataMediumToMedium 6598 6726 +1.9% 0.98x
DataAppendDataMediumToSmall 5946 6132 +3.1% 0.97x (?)
DataAppendDataSmallToLarge 37311 36980 -0.9% 1.01x (?)
DataAppendDataSmallToMedium 6330 6332 +0.0% 1.00x (?)
DataAppendDataSmallToSmall 5857 6027 +2.9% 0.97x (?)
DataCopyBytes 529 529 +0.0% 1.00x
DataMutateBytes 3932 3927 -0.1% 1.00x (?)
DataReplaceLarge 36811 36815 +0.0% 1.00x (?)
DataReplaceLargeBuffer 58968 58820 -0.3% 1.00x (?)
DataReplaceMedium 8049 7955 -1.2% 1.01x (?)
DataReplaceMediumBuffer 12719 12717 -0.0% 1.00x (?)
DataReplaceSmall 5938 5863 -1.3% 1.01x (?)
DataReset 2895 2925 +1.0% 0.99x (?)
DataSetCount 555 550 -0.9% 1.01x (?)
DataSubscript 220 220 +0.0% 1.00x
DictOfArraysToArrayOfDicts 835 821 -1.7% 1.02x (?)
Dictionary 519 522 +0.6% 0.99x
Dictionary2 659 657 -0.3% 1.00x
Dictionary2OfObjects 2097 2099 +0.1% 1.00x (?)
Dictionary3 223 224 +0.4% 1.00x
Dictionary3OfObjects 744 744 +0.0% 1.00x
Dictionary4 323 324 +0.3% 1.00x
Dictionary4Legacy 709 714 +0.7% 0.99x
Dictionary4OfObjects 439 437 -0.5% 1.00x (?)
Dictionary4OfObjectsLegacy 929 930 +0.1% 1.00x
DictionaryBridge 1222 1242 +1.6% 0.98x (?)
DictionaryBridgeToObjC_Bridge 19 19 +0.0% 1.00x
DictionaryBridgeToObjC_BulkAccess 166 167 +0.6% 0.99x (?)
DictionaryCompactMapValuesOfCastValue 14833 14664 -1.1% 1.01x (?)
DictionaryCompactMapValuesOfNilValue 7254 7408 +2.1% 0.98x
DictionaryCopy 108789 108926 +0.1% 1.00x (?)
DictionaryFilter 108857 109363 +0.5% 1.00x (?)
DictionaryGroup 215 216 +0.5% 1.00x
DictionaryGroupOfObjects 2122 2114 -0.4% 1.00x (?)
DictionaryKeysContainsCocoa 40 41 +2.5% 0.98x (?)
DictionaryKeysContainsNative 31 30 -3.2% 1.03x (?)
DictionaryLiteral 1943 1941 -0.1% 1.00x (?)
DictionaryOfObjects 2398 2384 -0.6% 1.01x (?)
DictionaryRemoveOfObjects 25670 26096 +1.7% 0.98x (?)
DictionarySubscriptDefaultMutation 269 267 -0.7% 1.01x
DictionarySubscriptDefaultMutationArray 623 620 -0.5% 1.00x
DictionarySubscriptDefaultMutationArrayOfObjects 4032 4038 +0.1% 1.00x (?)
DictionarySubscriptDefaultMutationOfObjects 1723 1716 -0.4% 1.00x
DictionarySwap 1048 1048 +0.0% 1.00x
DictionarySwapAtOfObjects 52354 52065 -0.6% 1.01x (?)
DictionarySwapOfObjects 8882 9093 +2.4% 0.98x
DoubleWidthDivision 0 0 +0.0% 1.00x
DropFirstAnyCollection 76 76 +0.0% 1.00x
DropFirstAnyCollectionLazy 65424 65254 -0.3% 1.00x (?)
DropFirstAnySeqCRangeIter 93 93 +0.0% 1.00x
DropFirstAnySeqCRangeIterLazy 93 93 +0.0% 1.00x
DropFirstAnySeqCntRange 71 71 +0.0% 1.00x
DropFirstAnySeqCntRangeLazy 71 71 +0.0% 1.00x
DropFirstAnySequence 1842 1843 +0.1% 1.00x (?)
DropFirstAnySequenceLazy 1845 1842 -0.2% 1.00x
DropFirstArray 35 35 +0.0% 1.00x
DropFirstArrayLazy 35 35 +0.0% 1.00x
DropFirstCountableRange 35 35 +0.0% 1.00x
DropFirstCountableRangeLazy 35 35 +0.0% 1.00x
DropFirstSequence 2683 2682 -0.0% 1.00x
DropFirstSequenceLazy 2771 2775 +0.1% 1.00x (?)
DropLastAnyCollection 28 28 +0.0% 1.00x
DropLastAnySeqCRangeIter 3360 3369 +0.3% 1.00x (?)
DropLastAnySeqCRangeIterLazy 3366 3370 +0.1% 1.00x (?)
DropLastAnySeqCntRange 11 11 +0.0% 1.00x
DropLastAnySeqCntRangeLazy 11 11 +0.0% 1.00x
DropLastAnySequence 4988 4960 -0.6% 1.01x
DropLastAnySequenceLazy 5075 5063 -0.2% 1.00x
DropWhileAnyCollection 100 100 +0.0% 1.00x
DropWhileAnyCollectionLazy 130 130 +0.0% 1.00x
DropWhileAnySeqCRangeIter 75 76 +1.3% 0.99x
DropWhileAnySeqCRangeIterLazy 130 130 +0.0% 1.00x
DropWhileAnySeqCntRange 95 95 +0.0% 1.00x
DropWhileAnySeqCntRangeLazy 130 130 +0.0% 1.00x
DropWhileAnySequence 1861 1861 +0.0% 1.00x
DropWhileAnySequenceLazy 1854 1854 +0.0% 1.00x
DropWhileArrayLazy 88 88 +0.0% 1.00x
DropWhileCountableRange 36 36 +0.0% 1.00x
DropWhileCountableRangeLazy 105 105 +0.0% 1.00x
DropWhileSequence 2218 2220 +0.1% 1.00x (?)
DropWhileSequenceLazy 88 88 +0.0% 1.00x
EqualStringSubstring 50 51 +2.0% 0.98x
EqualSubstringString 49 49 +0.0% 1.00x
EqualSubstringSubstring 49 49 +0.0% 1.00x
EqualSubstringSubstringGenericEquatable 49 49 +0.0% 1.00x
ErrorHandling 1213 1212 -0.1% 1.00x (?)
ExclusivityGlobal 5 5 +0.0% 1.00x
ExclusivityIndependent 2 2 +0.0% 1.00x
FatCompactMap 1246 1247 +0.1% 1.00x (?)
FilterEvenUsingReduce 1360 1336 -1.8% 1.02x (?)
FilterEvenUsingReduceInto 164 164 +0.0% 1.00x
FloatingPointPrinting_Double_description_small 21585 21554 -0.1% 1.00x (?)
FloatingPointPrinting_Double_description_uniform 21150 21143 -0.0% 1.00x (?)
FloatingPointPrinting_Double_interpolated 61222 61232 +0.0% 1.00x (?)
FloatingPointPrinting_Float80_description_small 28428 28362 -0.2% 1.00x (?)
FloatingPointPrinting_Float80_description_uniform 27741 27570 -0.6% 1.01x
FloatingPointPrinting_Float80_interpolated 64540 64513 -0.0% 1.00x (?)
FloatingPointPrinting_Float_description_small 5324 5384 +1.1% 0.99x
FloatingPointPrinting_Float_description_uniform 5226 5259 +0.6% 0.99x (?)
FloatingPointPrinting_Float_interpolated 37973 37839 -0.4% 1.00x (?)
FrequenciesUsingReduce 4979 5000 +0.4% 1.00x (?)
FrequenciesUsingReduceInto 1547 1546 -0.1% 1.00x (?)
HashTest 947 943 -0.4% 1.00x (?)
Histogram 640 647 +1.1% 0.99x
Integrate 337 349 +3.6% 0.97x
Join 157 157 +0.0% 1.00x
LazilyFilteredArrayContains 37471 36658 -2.2% 1.02x
LazilyFilteredArrays2 4635 4631 -0.1% 1.00x (?)
LazilyFilteredRange 3877 3745 -3.4% 1.04x
LessSubstringSubstring 48 48 +0.0% 1.00x
LessSubstringSubstringGenericComparable 48 48 +0.0% 1.00x
LinkedList 7612 7605 -0.1% 1.00x (?)
LuhnAlgoEager 452 458 +1.3% 0.99x (?)
LuhnAlgoLazy 443 449 +1.4% 0.99x
MapReduce 398 399 +0.3% 1.00x (?)
MapReduceAnyCollection 399 400 +0.3% 1.00x (?)
MapReduceAnyCollectionShort 2058 2058 +0.0% 1.00x
MapReduceClass 3006 3007 +0.0% 1.00x (?)
MapReduceClassShort 4613 4582 -0.7% 1.01x (?)
MapReduceLazyCollection 13 13 +0.0% 1.00x
MapReduceLazySequence 86 86 +0.0% 1.00x
MapReduceSequence 464 463 -0.2% 1.00x (?)
MapReduceShort 2051 2027 -1.2% 1.01x
MapReduceShortString 20 21 +5.0% 0.95x
Memset 218 217 -0.5% 1.00x (?)
MonteCarloE 10367 10293 -0.7% 1.01x
MonteCarloPi 42769 42743 -0.1% 1.00x
NSDictionaryCastToSwift 7366 7442 +1.0% 0.99x (?)
NSError 165 164 -0.6% 1.01x (?)
NSStringConversion 699 700 +0.1% 1.00x
NibbleSort 3447 3460 +0.4% 1.00x
ObjectAllocation 133 133 +0.0% 1.00x
ObjectiveCBridgeFromNSArrayAnyObject 26399 27127 +2.8% 0.97x (?)
ObjectiveCBridgeFromNSArrayAnyObjectForced 5393 5396 +0.1% 1.00x (?)
ObjectiveCBridgeFromNSArrayAnyObjectToString 48005 46538 -3.1% 1.03x (?)
ObjectiveCBridgeFromNSArrayAnyObjectToStringForced 45963 46449 +1.1% 0.99x (?)
ObjectiveCBridgeFromNSDictionaryAnyObject 113556 109067 -4.0% 1.04x (?)
ObjectiveCBridgeFromNSSetAnyObject 50413 49608 -1.6% 1.02x (?)
ObjectiveCBridgeFromNSSetAnyObjectForced 5058 5301 +4.8% 0.95x (?)
ObjectiveCBridgeFromNSSetAnyObjectToString 72567 71820 -1.0% 1.01x (?)
ObjectiveCBridgeFromNSString 1367 1382 +1.1% 0.99x
ObjectiveCBridgeFromNSStringForced 2822 2915 +3.3% 0.97x (?)
ObjectiveCBridgeStubDataAppend 6322 6292 -0.5% 1.00x (?)
ObjectiveCBridgeStubDateMutation 401 400 -0.2% 1.00x
ObjectiveCBridgeStubFromNSString 1048 1047 -0.1% 1.00x (?)
ObjectiveCBridgeStubNSDataAppend 2599 2592 -0.3% 1.00x (?)
ObjectiveCBridgeStubToNSString 2348 2341 -0.3% 1.00x (?)
ObjectiveCBridgeStubToNSStringRef 121 124 +2.5% 0.98x
ObjectiveCBridgeToNSArray 14578 14562 -0.1% 1.00x (?)
ObjectiveCBridgeToNSDictionary 27273 27900 +2.3% 0.98x (?)
ObjectiveCBridgeToNSString 452 452 +0.0% 1.00x
ObserverClosure 2156 2154 -0.1% 1.00x (?)
ObserverForwarderStruct 1205 1197 -0.7% 1.01x (?)
ObserverPartiallyAppliedMethod 3736 3718 -0.5% 1.00x (?)
ObserverUnappliedMethod 2470 2479 +0.4% 1.00x (?)
OpaqueConsumingUsers 4181 4181 +0.0% 1.00x
OpenClose 68 65 -4.4% 1.05x
Phonebook 7438 7313 -1.7% 1.02x
PolymorphicCalls 25 25 +0.0% 1.00x
PopFrontArray 1809 1828 +1.1% 0.99x
PopFrontArrayGeneric 1814 1837 +1.3% 0.99x
PopFrontUnsafePointer 8676 8680 +0.0% 1.00x (?)
PrefixAnyCollection 76 76 +0.0% 1.00x
PrefixAnyCollectionLazy 68144 65286 -4.2% 1.04x (?)
PrefixAnySeqCRangeIter 40 40 +0.0% 1.00x
PrefixAnySeqCRangeIterLazy 40 40 +0.0% 1.00x
PrefixAnySeqCntRange 71 71 +0.0% 1.00x
PrefixAnySeqCntRangeLazy 71 71 +0.0% 1.00x
PrefixAnySequence 1379 1378 -0.1% 1.00x (?)
PrefixAnySequenceLazy 1378 1379 +0.1% 1.00x (?)
PrefixArray 35 35 +0.0% 1.00x
PrefixArrayLazy 35 35 +0.0% 1.00x
PrefixCountableRange 35 35 +0.0% 1.00x
PrefixCountableRangeLazy 35 35 +0.0% 1.00x
PrefixSequence 2223 2215 -0.4% 1.00x
PrefixSequenceLazy 2274 2297 +1.0% 0.99x (?)
PrefixWhileAnyCollection 147 146 -0.7% 1.01x
PrefixWhileAnyCollectionLazy 71 71 +0.0% 1.00x
PrefixWhileAnySeqCRangeIter 405 389 -4.0% 1.04x
PrefixWhileAnySeqCRangeIterLazy 89 89 +0.0% 1.00x
PrefixWhileAnySeqCntRange 142 142 +0.0% 1.00x
PrefixWhileAnySequence 1542 1541 -0.1% 1.00x (?)
PrefixWhileAnySequenceLazy 1391 1391 +0.0% 1.00x
PrefixWhileArray 88 88 +0.0% 1.00x
PrefixWhileArrayLazy 70 70 +0.0% 1.00x
PrefixWhileSequence 380 364 -4.2% 1.04x
PrefixWhileSequenceLazy 52 52 +0.0% 1.00x
Prims 927 920 -0.8% 1.01x (?)
PrimsSplit 925 927 +0.2% 1.00x (?)
QueueConcrete 1153 1154 +0.1% 1.00x (?)
QueueGeneric 1140 1138 -0.2% 1.00x
RC4 151 151 +0.0% 1.00x
RGBHistogram 2597 2627 +1.2% 0.99x (?)
RGBHistogramOfObjects 20345 20344 -0.0% 1.00x (?)
Radix2CooleyTukey 12376 12482 +0.9% 0.99x (?)
Radix2CooleyTukeyf 9152 8958 -2.1% 1.02x (?)
RandomDoubleDef 27083 26974 -0.4% 1.00x
RandomDoubleLCG 2261 2203 -2.6% 1.03x
RandomIntegersDef 24510 24509 -0.0% 1.00x (?)
RandomIntegersLCG 173 173 +0.0% 1.00x
RandomShuffleLCG2 1808 1808 +0.0% 1.00x
RangeIterationSigned 200 200 +0.0% 1.00x
RangeReplaceableCollectionPlusDefault 1084 1077 -0.6% 1.01x (?)
RecursiveOwnedParameter 115 115 +0.0% 1.00x
RemoveWhereFilterInts 49 47 -4.1% 1.04x
RemoveWhereFilterString 249 247 -0.8% 1.01x
RemoveWhereFilterStrings 438 438 +0.0% 1.00x
RemoveWhereMoveInts 14 14 +0.0% 1.00x
RemoveWhereMoveStrings 710 708 -0.3% 1.00x (?)
RemoveWhereQuadraticInts 1295 1296 +0.1% 1.00x
RemoveWhereQuadraticString 376 373 -0.8% 1.01x
RemoveWhereQuadraticStrings 2762 2760 -0.1% 1.00x (?)
RemoveWhereSwapInts 20 20 +0.0% 1.00x
RemoveWhereSwapStrings 862 860 -0.2% 1.00x (?)
ReversedArray2 200 200 +0.0% 1.00x
ReversedBidirectional 14250 14057 -1.4% 1.01x
ReversedDictionary2 316 316 +0.0% 1.00x
RomanNumbers 84217 82969 -1.5% 1.02x (?)
SequenceAlgosAnySequence 12570 12539 -0.2% 1.00x (?)
SequenceAlgosArray 1574 1581 +0.4% 1.00x
SequenceAlgosContiguousArray 1579 1579 +0.0% 1.00x
SequenceAlgosList 1357 1358 +0.1% 1.00x (?)
SequenceAlgosRange 2576 2577 +0.0% 1.00x (?)
SequenceAlgosUnfoldSequence 1103 1102 -0.1% 1.00x
SetExclusiveOr 5135 5142 +0.1% 1.00x (?)
SetExclusiveOr_OfObjects 11695 11729 +0.3% 1.00x (?)
SetIntersect 695 696 +0.1% 1.00x (?)
SetIntersect_OfObjects 1775 1769 -0.3% 1.00x
SetIsSubsetOf 338 338 +0.0% 1.00x
SetIsSubsetOf_OfObjects 432 435 +0.7% 0.99x
SetUnion 4478 4377 -2.3% 1.02x
SetUnion_OfObjects 10197 10210 +0.1% 1.00x (?)
SevenBoom 837 847 +1.2% 0.99x (?)
Sim2DArray 417 417 +0.0% 1.00x
SortLargeExistentials 5538 5635 +1.8% 0.98x
SortLettersInPlace 961 960 -0.1% 1.00x (?)
SortSortedStrings 722 711 -1.5% 1.02x
SortStrings 1534 1501 -2.2% 1.02x
SortStringsUnicode 2168 2143 -1.2% 1.01x
StackPromo 24532 24539 +0.0% 1.00x (?)
StrComplexWalk 1774 1776 +0.1% 1.00x (?)
StrToInt 3430 3332 -2.9% 1.03x
StringAdder 551 551 +0.0% 1.00x
StringBuilder 496 497 +0.2% 1.00x
StringBuilderLong 1226 1220 -0.5% 1.00x (?)
StringBuilderSmallReservingCapacity 506 507 +0.2% 1.00x (?)
StringBuilderWithLongSubstring 1457 1471 +1.0% 0.99x (?)
StringComparison_abnormal 791 799 +1.0% 0.99x (?)
StringComparison_ascii 1047 1030 -1.6% 1.02x
StringComparison_emoji 858 854 -0.5% 1.00x (?)
StringComparison_fastPrenormal 881 860 -2.4% 1.02x
StringComparison_latin1 690 676 -2.0% 1.02x
StringComparison_longSharedPrefix 960 954 -0.6% 1.01x
StringComparison_nonBMPSlowestPrenormal 1689 1679 -0.6% 1.01x
StringComparison_slowerPrenormal 1828 1823 -0.3% 1.00x (?)
StringComparison_zalgo 112076 112409 +0.3% 1.00x
StringEdits 165597 165665 +0.0% 1.00x (?)
StringEnumRawValueInitialization 867 864 -0.3% 1.00x (?)
StringEqualPointerComparison 286 286 +0.0% 1.00x
StringFromLongWholeSubstring 21 21 +0.0% 1.00x
StringFromLongWholeSubstringGeneric 21 21 +0.0% 1.00x
StringHasPrefixAscii 2261 2233 -1.2% 1.01x
StringHasPrefixUnicode 99886 100054 +0.2% 1.00x (?)
StringHasSuffixAscii 2291 2262 -1.3% 1.01x
StringHasSuffixUnicode 100983 100665 -0.3% 1.00x (?)
StringHashing_abnormal 1345 1352 +0.5% 0.99x
StringHashing_ascii 35 35 +0.0% 1.00x
StringHashing_emoji 1896 1929 +1.7% 0.98x (?)
StringHashing_fastPrenormal 8234 8219 -0.2% 1.00x (?)
StringHashing_latin1 2520 2519 -0.0% 1.00x (?)
StringHashing_longSharedPrefix 7919 7893 -0.3% 1.00x
StringHashing_nonBMPSlowestPrenormal 2130 2132 +0.1% 1.00x (?)
StringHashing_slowerPrenormal 2692 2686 -0.2% 1.00x (?)
StringHashing_zalgo 3509 3506 -0.1% 1.00x (?)
StringInterpolation 8855 8923 +0.8% 0.99x (?)
StringInterpolationManySmallSegments 17892 17795 -0.5% 1.01x (?)
StringInterpolationSmall 4008 3971 -0.9% 1.01x
StringMatch 12284 12167 -1.0% 1.01x
StringRemoveDupes 489 494 +1.0% 0.99x
StringUTF16Builder 2543 2558 +0.6% 0.99x (?)
StringUTF16SubstringBuilder 5571 5651 +1.4% 0.99x (?)
StringWalk 1560 1547 -0.8% 1.01x
StringWithCString2 1976 1976 +0.0% 1.00x
StringWordBuilder 2428 2432 +0.2% 1.00x (?)
StringWordBuilderReservingCapacity 1632 1640 +0.5% 1.00x (?)
SubstringComparable 13 13 +0.0% 1.00x
SubstringEqualString 621 617 -0.6% 1.01x (?)
SubstringEquatable 1432 1435 +0.2% 1.00x (?)
SubstringFromLongString 10 10 +0.0% 1.00x
SubstringFromLongStringGeneric 74 74 +0.0% 1.00x
SuffixAnyCollection 28 28 +0.0% 1.00x
SuffixAnyCollectionLazy 21819 21795 -0.1% 1.00x (?)
SuffixAnySeqCRangeIter 3666 3659 -0.2% 1.00x
SuffixAnySeqCRangeIterLazy 3662 3658 -0.1% 1.00x (?)
SuffixAnySeqCntRange 17 17 +0.0% 1.00x
SuffixAnySeqCntRangeLazy 17 17 +0.0% 1.00x
SuffixAnySequence 4944 4943 -0.0% 1.00x (?)
SuffixAnySequenceLazy 5070 5070 +0.0% 1.00x
SuffixSequence 3669 3667 -0.1% 1.00x (?)
SuffixSequenceLazy 3670 3664 -0.2% 1.00x (?)
SumUsingReduce 102 101 -1.0% 1.01x
SumUsingReduceInto 97 97 +0.0% 1.00x
SuperChars 19655 19536 -0.6% 1.01x (?)
TwoSum 1504 1501 -0.2% 1.00x (?)
TypeFlood 0 0 +0.0% 1.00x
UTF8Decode 298 299 +0.3% 1.00x
UTF8Decode_InitDecoding 1363 1365 +0.1% 1.00x
UTF8Decode_InitDecoding_ascii 654 656 +0.3% 1.00x (?)
UTF8Decode_InitFromData 1338 1370 +2.4% 0.98x (?)
Walsh 428 427 -0.2% 1.00x (?)
WordCountHistogramASCII 7123 7019 -1.5% 1.01x
WordCountHistogramUTF16 10464 10530 +0.6% 0.99x (?)
WordCountUniqueASCII 2089 2134 +2.2% 0.98x
WordCountUniqueUTF16 4818 4649 -3.5% 1.04x (?)
XorLoop 404 404 +0.0% 1.00x
Added (11)
TEST MIN MAX MEAN MAX_RSS
Chars2 2527 2529 2528 8560640
DropLastCountableRange2 1183 1183 1183 8570197
DropLastCountableRangeLazy2 1203 1204 1203 8566101
ObjectiveCBridgeStubFromArrayOfNSString2 4350 4678 4474 9622869
ObjectiveCBridgeStubToArrayOfNSString2 4069 4335 4165 9641984
ObjectiveCBridgeStubToNSDate2 1606 1706 1646 9614677
ObjectiveCBridgeStubURLAppendPath2 2809 3057 2947 8699904
ObjectiveCBridgeStubURLAppendPathRef2 2916 2938 2924 8687616
PrefixWhileAnySeqCntRangeLazy2 715 716 715 8626176
SuffixCountableRange2 1182 1188 1184 8566101
SuffixCountableRangeLazy2 1183 1184 1183 8575659
Removed (19)
TEST MIN MAX MEAN MAX_RSS
Chars 917 919 918 8568832
DropLastCountableRange 11 12 12 8563371
DropLastCountableRangeLazy 11 11 11 8570197
ObjectiveCBridgeStubFromArrayOfNSString 37573 42035 39958 9647445
ObjectiveCBridgeStubFromNSDate 6588 7795 6993 9628331
ObjectiveCBridgeStubFromNSStringRef 171 171 171 9606485
ObjectiveCBridgeStubNSDateMutationRef 13264 15267 14129 8572928
ObjectiveCBridgeStubToArrayOfNSString 39555 45480 42098 9678848
ObjectiveCBridgeStubToNSDate 15129 17660 16202 9565525
ObjectiveCBridgeStubToNSDateRef 3420 3508 3475 9564160
ObjectiveCBridgeStubURLAppendPath 284736 298904 290923 8695808
ObjectiveCBridgeStubURLAppendPathRef 272852 294889 284477 8712192
PrefixWhileAnySeqCntRangeLazy 71 71 71 8622080
PrefixWhileCountableRange 36 36 36 8571563
PrefixWhileCountableRangeLazy 35 35 35 8567467
SuffixCountableRange 11 11 11 8568832
SuffixCountableRangeLazy 11 11 11 8572928
WordSplitASCII 9671 10618 10023 8915627
WordSplitUTF16 11097 11803 11369 8903339
Hardware Overview 
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

eeckstein added 4 commits July 31, 2018 10:59
@eeckstein
benchmarks: fix the Calculator benchmark by covering more String comp…
2839c17 
…are test cases
@eeckstein
benchmarks: fix the Chars benchmark by adding blackHoles and testing …
30cc37e 
…all possible Char comparisons
@eeckstein
benchmarks: Marking some benchmarks as unstable
868c5a1 
The noise for those benchmarks even cannot be removed by using a high sample count.
@eeckstein
benchmarks: fix the iteration count for some benchmarks
7299894 
So that a single iteration is within ~2ms (and also not too short).
@eeckstein
Copy link
Contributor Author

@swift-ci smoke test and merge

1 similar comment
@eeckstein
Copy link
Contributor Author

@swift-ci smoke test and merge

@swift-ci swift-ci merged commit 8144728 into swiftlang:master Jul 31, 2018
@eeckstein eeckstein deleted the fix-benchmarks2 branch July 31, 2018 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@eeckstein @swift-ci