[WIP][Compare Perf] Add support for markdown and html #1682

shahmishal · 2016-03-14T21:22:52Z

What's in this pull request?

Compare perf results supports following args:

optional arguments:
  -h, --help            show this help message and exit
  --old-file OLD_FILE   Baseline performance test suite (csv file)
  --new-file NEW_FILE   New performance test suite (csv file)
  --format FORMAT       Supported format git, html and markdown
  --output OUTPUT       Output file name
  --changes-only        Output only affected tests
  --new-branch NEW_BRANCH
                        Name of the new branch
  --old-branch OLD_BRANCH
                        Name of the old branch
  --delta-threshold DELTA_THRESHOLD
                        delta threshold

Resolved bug number: N/A

Before merging this pull request to apple/swift repository:

Test pull request on Swift continuous integration.

Triggering Swift CI

The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:

Smoke Testing

Platform	Comment
All supported platforms	@swift-ci Please smoke test
OS X platform	@swift-ci Please smoke test OS X platform
Linux platform	@swift-ci Please smoke test Linux platform

Validation Testing

Platform	Comment
All supported platforms	@swift-ci Please test
OS X platform	@swift-ci Please test OS X platform
Linux platform	@swift-ci Please test Linux platform

Note: Only members of the Apple organization can trigger swift-ci.

shahmishal · 2016-03-14T21:23:06Z

@swift-ci Please perf test

practicalswift · 2016-03-14T21:36:10Z

@shahmishal Nice stuff!

A friendly side note - I think there might be a few flake8 regressions in there that might be worth looking in to:

$ flake8 | cut -f4- -d: | sort | uniq -c
    126  E111 indentation is not a multiple of four
      3  E122 continuation line missing indentation or outdented
      3  E127 continuation line over-indented for visual indent
      3  E128 continuation line under-indented for visual indent
      1  E131 continuation line unaligned for hanging indent
      1  E203 whitespace before ':'
      3  E222 multiple spaces after operator
      3  E231 missing whitespace after ','
      6  E302 expected 2 blank lines, found 1
     16  E501 line too long (… > 79 characters)
      2  E711 comparison to None should be 'if cond is not None:'
      1  F401 'os' imported but unused
      1  F841 local variable 'geogebra_value' is assigned to but never used
      1  F841 local variable 'normalList' is assigned to but never used
      1  I100 Import statements are in the wrong order. import argparse should be before import csv
      1  I100 Import statements are in the wrong order. import csv should be before import sys
      1  N806 variable in function should be lowercase

I can fix them afterwards but that will cause some unnecessary git blame noise :-)

shahmishal · 2016-03-14T21:41:12Z

@practicalswift Thanks I will fix it :)

gottesmm · 2016-03-14T22:20:13Z

@shahmishal I wonder if we can make flake8 testing a lit test...

gottesmm · 2016-03-14T22:21:15Z

@practicalswift Thanks for the command line. I think we can probably make this into a lit test. We could make cmake search for flake8 and enable it only if we find flake8.

shahmishal · 2016-03-14T23:40:28Z

@swift-ci Please perf test

shahmishal · 2016-03-15T05:50:18Z

@swift-ci Please perf test

swift-ci · 2016-03-15T08:30:47Z

!!! Couldn't read commit file !!!

practicalswift · 2016-03-15T09:01:58Z

@shahmishal Nice! flake8 gives 👍 now :-)

When running with the extension flake8-import-order enabled it gives the following warning about imports not being in the recommended order:

./benchmark/scripts/compare_perf_tests.py:17:1: I100 Import statements are in the wrong order. import csv should be before import sys
./benchmark/scripts/compare_perf_tests.py:18:1: I100 Import statements are in the wrong order. import argparse should be before import csv

shahmishal · 2016-03-22T01:15:56Z

@swift-ci Please perf test

swift-ci · 2016-03-22T04:01:02Z

Build comment file:

** O **

Regression (4)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
ObjectiveCBridgeStubFromNSString	780	882	+13.1%	0.88x(?)
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	124694	136775	+9.7%	0.91x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced	150795	161812	+7.3%	0.93x(?)
ObjectiveCBridgeStubFromArrayOfNSString	141945	150536	+6.0%	0.94x(?)

Improvement (3)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
ObjectiveCBridgeFromNSArrayAnyObjectToStringForced	209548	195499	-6.7%	1.07x
ObjectiveCBridgeFromNSArrayAnyObjectForced	4208	3842	-8.7%	1.10x(?)
Histogram	848	750	-11.6%	1.13x

No Changes (107)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
DictionaryLiteral	3495	3332	-4.7%	1.05x(?)
NSDictionaryCastToSwift	12001	11477	-4.4%	1.05x(?)
ObjectAllocation	159	153	-3.8%	1.04x(?)
StrComplexWalk	3568	3419	-4.2%	1.04x(?)
StaticArray	3012	2934	-2.6%	1.03x(?)
ObjectiveCBridgeFromNSArrayAnyObject	73690	71484	-3.0%	1.03x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectForced	5007	4908	-2.0%	1.02x(?)
Dictionary3	1254	1235	-1.5%	1.02x(?)
ObjectiveCBridgeFromNSSetAnyObjectToString	223929	219376	-2.0%	1.02x(?)
RGBHistogram	3351	3279	-2.1%	1.02x(?)
ArrayOfPOD	185	182	-1.6%	1.02x(?)
ArraySubscript	1423	1405	-1.3%	1.01x(?)
ErrorHandling	2855	2827	-1.0%	1.01x(?)
ObjectiveCBridgeFromNSSetAnyObject	134661	133325	-1.0%	1.01x(?)
Dictionary2	3496	3449	-1.3%	1.01x(?)
ArrayOfGenericRef	3593	3556	-1.0%	1.01x(?)
SetUnion	3772	3744	-0.7%	1.01x(?)
ObjectiveCBridgeToNSString	1058	1059	+0.1%	1.00x(?)
DictionarySwapOfObjects	9039	9002	-0.4%	1.00x(?)
StackPromo	20266	20262	-0.0%	1.00x(?)
PopFrontArray	1997	1999	+0.1%	1.00x
RecursiveOwnedParameter	1987	1986	-0.1%	1.00x
ObjectiveCBridgeStubToNSString	1385	1386	+0.1%	1.00x
ClassArrayGetter	15	15	+0.0%	1.00x
Array2D	2812	2805	-0.2%	1.00x
SortStrings	1867	1868	+0.1%	1.00x(?)
OpenClose	68	68	+0.0%	1.00x
StringWithCString	80916	81029	+0.1%	1.00x(?)
ObjectiveCBridgeFromNSStringForced	2394	2393	-0.0%	1.00x(?)
Prims	754	752	-0.3%	1.00x(?)
SortLettersInPlace	1061	1060	-0.1%	1.00x(?)
DictionarySwap	801	802	+0.1%	1.00x(?)
Dictionary3OfObjects	1834	1838	+0.2%	1.00x
RangeAssignment	321	321	+0.0%	1.00x
ByteSwap	1	1	+0.0%	1.00x
SuperChars	589072	591101	+0.3%	1.00x(?)
StringInterpolation	11220	11263	+0.4%	1.00x(?)
Integrate	249	249	+0.0%	1.00x
ProtocolDispatch	3284	3284	+0.0%	1.00x
TypeFlood	0	0	+0.0%	1.00x
ArrayLiteral	1013	1013	+0.0%	1.00x
ProtocolDispatch2	147	147	+0.0%	1.00x
SetIntersect_OfObjects	3506	3501	-0.1%	1.00x(?)
Join	500	500	+0.0%	1.00x
ArrayOfRef	3525	3526	+0.0%	1.00x(?)
NSError	363	362	-0.3%	1.00x(?)
DictionaryOfObjects	4166	4166	+0.0%	1.00x
PopFrontUnsafePointer	8714	8723	+0.1%	1.00x
CaptureProp	5740	5739	-0.0%	1.00x(?)
PolymorphicCalls	102	102	+0.0%	1.00x
Calculator	45	45	+0.0%	1.00x
MapReduce	417	417	+0.0%	1.00x
Hanoi	3726	3723	-0.1%	1.00x(?)
DictionaryRemoveOfObjects	20254	20183	-0.3%	1.00x
UTF8Decode	908	907	-0.1%	1.00x(?)
SetIsSubsetOf	513	512	-0.2%	1.00x
Dictionary	1960	1955	-0.3%	1.00x(?)
NopDeinit	35331	35330	-0.0%	1.00x(?)
ObjectiveCBridgeStubToArrayOfNSString	31067	31114	+0.1%	1.00x(?)
SetIntersect	1105	1104	-0.1%	1.00x(?)
SetExclusiveOr	4089	4108	+0.5%	1.00x(?)
RGBHistogramOfObjects	25988	25926	-0.2%	1.00x
StrToInt	4430	4426	-0.1%	1.00x(?)
ArrayInClass	91	91	+0.0%	1.00x
ObjectiveCBridgeFromNSString	1653	1647	-0.4%	1.00x(?)
Phonebook	11770	11766	-0.0%	1.00x(?)
Sim2DArray	793	791	-0.2%	1.00x(?)
SetExclusiveOr_OfObjects	13825	13794	-0.2%	1.00x(?)
MonteCarloPi	44212	44215	+0.0%	1.00x(?)
MonteCarloE	10479	10481	+0.0%	1.00x(?)
SetUnion_OfObjects	11217	11243	+0.2%	1.00x(?)
GlobalClass	0	0	+0.0%	1.00x
Walsh	360	360	+0.0%	1.00x
SetIsSubsetOf_OfObjects	671	671	+0.0%	1.00x
ArrayAppend	766	767	+0.1%	1.00x(?)
DictionaryRemove	5185	5173	-0.2%	1.00x(?)
LinkedList	9704	9695	-0.1%	1.00x(?)
NSStringConversion	637	636	-0.2%	1.00x
ArrayValueProp4	8	8	+0.0%	1.00x
Chars	895	894	-0.1%	1.00x(?)
StringBuilder	1277	1281	+0.3%	1.00x(?)
DeadArray	172	172	+0.0%	1.00x
BitCount	19	19	+0.0%	1.00x
AngryPhonebook	0	0	+0.0%	1.00x
SevenBoom	1361	1362	+0.1%	1.00x(?)
StringWalk	5166	5166	+0.0%	1.00x
ArrayValueProp	8	8	+0.0%	1.00x
RC4	238	238	+0.0%	1.00x
Memset	213	212	-0.5%	1.00x
Dictionary2OfObjects	5344	5356	+0.2%	1.00x(?)
ArrayAppendReserved	517	516	-0.2%	1.00x(?)
ArrayValueProp2	8	8	+0.0%	1.00x
ArrayValueProp3	8	8	+0.0%	1.00x
DictionaryBridge	5533	5594	+1.1%	0.99x(?)
XorLoop	400	406	+1.5%	0.99x
113	3161354	3196477	+1.1%	0.99x
ObjectiveCBridgeToNSArray	31011	31405	+1.3%	0.99x(?)
PopFrontArrayGeneric	1982	1997	+0.8%	0.99x(?)
ObjectiveCBridgeFromNSDictionaryAnyObject	276924	281321	+1.6%	0.98x(?)
HashTest	1702	1729	+1.6%	0.98x
ArrayOfGenericPOD	219	223	+1.8%	0.98x
TwoSum	1701	1730	+1.7%	0.98x(?)
ObjectiveCBridgeToNSSet	24575	25343	+3.1%	0.97x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToString	276332	286497	+3.7%	0.96x
ObjectiveCBridgeFromNSArrayAnyObjectToString	186536	194354	+4.2%	0.96x(?)
ObjectiveCBridgeToNSDictionary	24404	25297	+3.7%	0.96x(?)
ObjectiveCBridgeFromNSSetAnyObjectForced	4376	4548	+3.9%	0.96x(?)

*\* Onone **

Regression (2)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
ObjectiveCBridgeFromNSString	5491	6084	+10.8%	0.90x(?)
Phonebook	60139	64946	+8.0%	0.93x(?)

Improvement (5)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
ObjectiveCBridgeFromNSSetAnyObject	145176	136777	-5.8%	1.06x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced	168723	158977	-5.8%	1.06x(?)
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	150019	140918	-6.1%	1.06x(?)
StringBuilder	9295	8668	-6.8%	1.07x(?)
NopDeinit	107341	98261	-8.5%	1.09x

No Changes (107)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
NSError	677	642	-5.2%	1.05x(?)
ObjectiveCBridgeFromNSArrayAnyObject	79645	75769	-4.9%	1.05x
Dictionary2OfObjects	9139	8663	-5.2%	1.05x(?)
Dictionary	5913	5712	-3.4%	1.04x
NSDictionaryCastToSwift	13739	13250	-3.6%	1.04x(?)
StackPromo	86389	84196	-2.5%	1.03x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToString	280632	272599	-2.9%	1.03x(?)
DictionaryBridge	5768	5620	-2.6%	1.03x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectForced	9210	8936	-3.0%	1.03x(?)
ObjectAllocation	1417	1375	-3.0%	1.03x(?)
SetIntersect	21347	20796	-2.6%	1.03x(?)
SetExclusiveOr_OfObjects	75174	72699	-3.3%	1.03x(?)
ArraySubscript	5037	4956	-1.6%	1.02x(?)
ObjectiveCBridgeFromNSArrayAnyObjectForced	8081	7918	-2.0%	1.02x(?)
HashTest	6162	6059	-1.7%	1.02x(?)
DictionaryLiteral	21278	20828	-2.1%	1.02x(?)
DeadArray	114909	113070	-1.6%	1.02x(?)
ObjectiveCBridgeStubFromArrayOfNSString	140036	136722	-2.4%	1.02x(?)
ObjectiveCBridgeFromNSArrayAnyObjectToString	188676	186950	-0.9%	1.01x(?)
113	6202260	6148289	-0.9%	1.01x
StaticArray	20913	20777	-0.7%	1.01x(?)
ErrorHandling	7812	7759	-0.7%	1.01x(?)
ObjectiveCBridgeToNSArray	30710	30398	-1.0%	1.01x(?)
DictionaryRemoveOfObjects	71525	71154	-0.5%	1.01x(?)
ObjectiveCBridgeToNSSet	28311	27924	-1.4%	1.01x(?)
RGBHistogram	56756	55927	-1.5%	1.01x(?)
BitCount	102	101	-1.0%	1.01x
ObjectiveCBridgeFromNSArrayAnyObjectToStringForced	203271	201657	-0.8%	1.01x(?)
ArrayValueProp2	19440	19180	-1.3%	1.01x(?)
ObjectiveCBridgeToNSString	1095	1093	-0.2%	1.00x(?)
DictionarySwapOfObjects	28624	28649	+0.1%	1.00x(?)
PopFrontArray	15062	15018	-0.3%	1.00x(?)
RecursiveOwnedParameter	11042	11007	-0.3%	1.00x(?)
Integrate	368	367	-0.3%	1.00x(?)
ClassArrayGetter	1351	1351	+0.0%	1.00x
Array2D	823840	824285	+0.1%	1.00x(?)
SortStrings	6029	6028	-0.0%	1.00x(?)
MonteCarloPi	53541	53473	-0.1%	1.00x(?)
StringWithCString	348239	348320	+0.0%	1.00x(?)
SortLettersInPlace	2376	2377	+0.0%	1.00x(?)
DictionarySwap	12105	12096	-0.1%	1.00x(?)
ArrayOfGenericPOD	4034	4046	+0.3%	1.00x(?)
ByteSwap	9	9	+0.0%	1.00x
SuperChars	769738	771601	+0.2%	1.00x(?)
XorLoop	19601	19601	+0.0%	1.00x
ObjectiveCBridgeStubToNSString	1718	1715	-0.2%	1.00x(?)
Join	1400	1399	-0.1%	1.00x(?)
ObjectiveCBridgeStubFromNSString	1234	1236	+0.2%	1.00x(?)
ProtocolDispatch	6026	6028	+0.0%	1.00x(?)
ArrayLiteral	1097	1093	-0.4%	1.00x(?)
ProtocolDispatch2	454	453	-0.2%	1.00x
StrComplexWalk	9219	9187	-0.3%	1.00x(?)
SetIntersect_OfObjects	20375	20367	-0.0%	1.00x(?)
ArrayOfRef	10668	10685	+0.2%	1.00x(?)
ArrayAppend	3262	3258	-0.1%	1.00x(?)
PopFrontUnsafePointer	108339	108298	-0.0%	1.00x(?)
CaptureProp	55595	55600	+0.0%	1.00x(?)
PolymorphicCalls	1135	1130	-0.4%	1.00x
Calculator	1049	1049	+0.0%	1.00x
MapReduce	46549	46502	-0.1%	1.00x(?)
Hanoi	24507	24536	+0.1%	1.00x(?)
OpenClose	440	442	+0.5%	1.00x(?)
UTF8Decode	40921	41019	+0.2%	1.00x(?)
SetExclusiveOr	53156	53347	+0.4%	1.00x(?)
ObjectiveCBridgeFromNSSetAnyObjectToString	230857	231411	+0.2%	1.00x(?)
StrToInt	18673	18679	+0.0%	1.00x(?)
ArrayInClass	3904	3904	+0.0%	1.00x
ArrayOfGenericRef	11508	11513	+0.0%	1.00x(?)
Sim2DArray	14062	14055	-0.1%	1.00x(?)
MonteCarloE	99820	100042	+0.2%	1.00x
SetUnion_OfObjects	54878	55017	+0.2%	1.00x(?)
RC4	9345	9340	-0.1%	1.00x(?)
Walsh	13631	13632	+0.0%	1.00x(?)
SetIsSubsetOf_OfObjects	3079	3075	-0.1%	1.00x(?)
DictionaryRemove	32545	32568	+0.1%	1.00x(?)
LinkedList	27620	27593	-0.1%	1.00x(?)
ArrayAppendReserved	3007	3011	+0.1%	1.00x
ArrayOfPOD	2317	2319	+0.1%	1.00x(?)
SetUnion	38097	38150	+0.1%	1.00x(?)
Chars	57083	57110	+0.1%	1.00x(?)
AngryPhonebook	3619	3622	+0.1%	1.00x(?)
ArrayValueProp3	2746	2740	-0.2%	1.00x(?)
StringWalk	23219	23306	+0.4%	1.00x(?)
ArrayValueProp	2344	2350	+0.3%	1.00x(?)
GlobalClass	0	0	+0.0%	1.00x
Memset	20608	20592	-0.1%	1.00x(?)
ArrayValueProp4	2693	2688	-0.2%	1.00x(?)
SevenBoom	1483	1488	+0.3%	1.00x(?)
Histogram	14964	15163	+1.3%	0.99x(?)
ObjectiveCBridgeFromNSStringForced	2801	2818	+0.6%	0.99x(?)
Prims	14934	15109	+1.2%	0.99x(?)
RangeAssignment	16508	16594	+0.5%	0.99x(?)
TypeFlood	148	150	+1.4%	0.99x(?)
Dictionary3	2761	2799	+1.4%	0.99x(?)
PopFrontArrayGeneric	15188	15413	+1.5%	0.99x(?)
ObjectiveCBridgeStubToArrayOfNSString	33134	33467	+1.0%	0.99x(?)
TwoSum	11156	11305	+1.3%	0.99x
StringInterpolation	15042	15332	+1.9%	0.98x
Dictionary2	6955	7073	+1.7%	0.98x(?)
ObjectiveCBridgeToNSDictionary	25373	26012	+2.5%	0.98x(?)
RGBHistogramOfObjects	107100	109329	+2.1%	0.98x(?)
NSStringConversion	3074	3152	+2.5%	0.98x(?)
Dictionary3OfObjects	3704	3815	+3.0%	0.97x(?)
ObjectiveCBridgeFromNSSetAnyObjectForced	8747	9055	+3.5%	0.97x(?)
ObjectiveCBridgeFromNSDictionaryAnyObject	284039	295493	+4.0%	0.96x(?)
DictionaryOfObjects	9180	9618	+4.8%	0.95x(?)
SetIsSubsetOf	3254	3434	+5.5%	0.95x(?)

shahmishal · 2016-03-22T17:59:47Z

@swift-ci Please benchmark

swift-ci · 2016-03-22T18:57:51Z

Build comment file:

Optimized (O)

Regression (0)

Improvement (0)

No Changes (114)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
DictionaryLiteral	3462	3370	-2.7%	1.03x(?)
ErrorHandling	2896	2843	-1.8%	1.02x(?)
NSDictionaryCastToSwift	11760	11543	-1.9%	1.02x
SortLettersInPlace	1077	1071	-0.6%	1.01x(?)
SuperChars	602239	595560	-1.1%	1.01x(?)
StaticArray	2868	2829	-1.4%	1.01x(?)
ObjectiveCBridgeToNSArray	30602	30172	-1.4%	1.01x
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced	129333	128106	-0.9%	1.01x(?)
SetExclusiveOr_OfObjects	14069	13995	-0.5%	1.01x(?)
ObjectiveCBridgeToNSDictionary	19122	18894	-1.2%	1.01x(?)
DictionaryRemove	5312	5282	-0.6%	1.01x(?)
TwoSum	1740	1729	-0.6%	1.01x(?)
ArraySubscript	1432	1437	+0.3%	1.00x(?)
ObjectiveCBridgeToNSString	1070	1073	+0.3%	1.00x(?)
DictionarySwapOfObjects	8975	8975	+0.0%	1.00x
StackPromo	20056	20070	+0.1%	1.00x
RecursiveOwnedParameter	1937	1938	+0.1%	1.00x(?)
ObjectiveCBridgeStubToNSString	1444	1442	-0.1%	1.00x(?)
ClassArrayGetter	13	13	+0.0%	1.00x
Array2D	2087	2083	-0.2%	1.00x(?)
Histogram	753	753	+0.0%	1.00x
SortStrings	1747	1746	-0.1%	1.00x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectForced	4767	4770	+0.1%	1.00x(?)
MonteCarloPi	45185	45197	+0.0%	1.00x
StringWithCString	80556	80591	+0.0%	1.00x(?)
ObjectiveCBridgeFromNSArrayAnyObjectForced	3802	3802	+0.0%	1.00x
ObjectiveCBridgeFromNSStringForced	2256	2263	+0.3%	1.00x(?)
Prims	779	777	-0.3%	1.00x(?)
DictionarySwap	784	785	+0.1%	1.00x
PopFrontArray	1304	1304	+0.0%	1.00x
ArrayOfGenericPOD	220	220	+0.0%	1.00x
Dictionary3OfObjects	1856	1861	+0.3%	1.00x(?)
RangeAssignment	275	276	+0.4%	1.00x(?)
ByteSwap	1	1	+0.0%	1.00x
XorLoop	380	380	+0.0%	1.00x
StringInterpolation	10965	10976	+0.1%	1.00x(?)
Integrate	238	238	+0.0%	1.00x
113	2750241	2760795	+0.4%	1.00x
ObjectiveCBridgeStubFromNSString	739	740	+0.1%	1.00x(?)
ProtocolDispatch	3040	3042	+0.1%	1.00x(?)
ObjectAllocation	149	149	+0.0%	1.00x
TypeFlood	0	0	+0.0%	1.00x
ObjectiveCBridgeFromNSSetAnyObject	119417	119577	+0.1%	1.00x(?)
AngryPhonebook	0	0	+0.0%	1.00x
ProtocolDispatch2	156	156	+0.0%	1.00x
Walsh	322	323	+0.3%	1.00x
Dictionary3	1275	1269	-0.5%	1.00x
Dictionary2	3533	3542	+0.2%	1.00x(?)
SetIntersect_OfObjects	3453	3455	+0.1%	1.00x(?)
Join	487	489	+0.4%	1.00x
ObjectiveCBridgeToNSSet	18757	18763	+0.0%	1.00x(?)
NSError	375	375	+0.0%	1.00x
DictionaryOfObjects	4206	4204	-0.1%	1.00x(?)
PopFrontArrayGeneric	1292	1291	-0.1%	1.00x(?)
PopFrontUnsafePointer	9004	9006	+0.0%	1.00x(?)
CaptureProp	4864	4864	+0.0%	1.00x
PolymorphicCalls	91	91	+0.0%	1.00x
Calculator	38	38	+0.0%	1.00x
MapReduce	341	342	+0.3%	1.00x(?)
Hanoi	3472	3479	+0.2%	1.00x(?)
OpenClose	51	51	+0.0%	1.00x
DictionaryRemoveOfObjects	20225	20287	+0.3%	1.00x
UTF8Decode	897	897	+0.0%	1.00x
SetIsSubsetOf	490	490	+0.0%	1.00x
Dictionary	1964	1964	+0.0%	1.00x
NopDeinit	35144	35124	-0.1%	1.00x(?)
SetIntersect	1119	1118	-0.1%	1.00x
SetExclusiveOr	4040	4029	-0.3%	1.00x(?)
RGBHistogramOfObjects	26095	26084	-0.0%	1.00x(?)
StrToInt	3913	3910	-0.1%	1.00x(?)
ArrayOfGenericRef	3620	3632	+0.3%	1.00x(?)
ObjectiveCBridgeFromNSString	1601	1600	-0.1%	1.00x(?)
Phonebook	11215	11269	+0.5%	1.00x(?)
Sim2DArray	633	632	-0.2%	1.00x
MonteCarloE	10661	10654	-0.1%	1.00x
SetUnion_OfObjects	11243	11225	-0.2%	1.00x
GlobalClass	0	0	+0.0%	1.00x
HashTest	1683	1687	+0.2%	1.00x(?)
SetIsSubsetOf_OfObjects	637	637	+0.0%	1.00x
ArrayAppend	770	770	+0.0%	1.00x
LinkedList	9519	9524	+0.1%	1.00x(?)
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	114301	113768	-0.5%	1.00x(?)
NSStringConversion	615	614	-0.2%	1.00x
RGBHistogram	3217	3203	-0.4%	1.00x(?)
ObjectiveCBridgeFromNSSetAnyObjectForced	4242	4235	-0.2%	1.00x(?)
ArrayValueProp4	6	6	+0.0%	1.00x
ArrayOfPOD	182	182	+0.0%	1.00x
SetUnion	3713	3708	-0.1%	1.00x(?)
Chars	835	835	+0.0%	1.00x
StringBuilder	1297	1298	+0.1%	1.00x(?)
DeadArray	182	182	+0.0%	1.00x
BitCount	16	16	+0.0%	1.00x
ArrayLiteral	1023	1023	+0.0%	1.00x
SevenBoom	1365	1359	-0.4%	1.00x(?)
ObjectiveCBridgeFromNSArrayAnyObjectToStringForced	166240	165582	-0.4%	1.00x(?)
StringWalk	5961	5962	+0.0%	1.00x(?)
ArrayValueProp	6	6	+0.0%	1.00x
RC4	251	251	+0.0%	1.00x
Memset	235	236	+0.4%	1.00x
Dictionary2OfObjects	5384	5370	-0.3%	1.00x(?)
ArrayAppendReserved	537	537	+0.0%	1.00x
ArrayValueProp2	6	6	+0.0%	1.00x
ArrayValueProp3	6	6	+0.0%	1.00x
ObjectiveCBridgeFromNSDictionaryAnyObject	235784	239280	+1.5%	0.99x(?)
DictionaryBridge	3862	3883	+0.5%	0.99x(?)
StrComplexWalk	2834	2852	+0.6%	0.99x(?)
ArrayOfRef	3567	3593	+0.7%	0.99x(?)
ObjectiveCBridgeFromNSArrayAnyObject	65998	66444	+0.7%	0.99x(?)
ArrayInClass	86	87	+1.2%	0.99x(?)
ObjectiveCBridgeFromNSSetAnyObjectToString	191251	192248	+0.5%	0.99x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToString	205329	208641	+1.6%	0.98x(?)
ObjectiveCBridgeStubToArrayOfNSString	30058	30741	+2.3%	0.98x
ObjectiveCBridgeStubFromArrayOfNSString	112158	113951	+1.6%	0.98x(?)
ObjectiveCBridgeFromNSArrayAnyObjectToString	162914	171781	+5.4%	0.95x(?)

**Unoptimized (Onone)**

Regression (1)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
TypeFlood	148	161	+8.8%	0.92x(?)

Improvement (2)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
StringWalk	35167	32312	-8.1%	1.09x
Array2D	841703	753200	-10.5%	1.12x(?)

No Changes (111)

TEST	OLD_MIN	NEW_MIN	DELTA (%)	SPEEDUP
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced	135419	131355	-3.0%	1.03x(?)
Phonebook	64404	62432	-3.1%	1.03x(?)
ObjectiveCBridgeFromNSSetAnyObjectToString	208090	202529	-2.7%	1.03x(?)
ObjectiveCBridgeFromNSArrayAnyObjectToStringForced	168457	165418	-1.8%	1.02x
ObjectiveCBridgeFromNSArrayAnyObjectToString	168836	166820	-1.2%	1.01x(?)
ObjectiveCBridgeFromNSDictionaryAnyObjectForced	7419	7381	-0.5%	1.01x(?)
ObjectiveCBridgeFromNSArrayAnyObjectForced	7427	7382	-0.6%	1.01x(?)
ObjectiveCBridgeFromNSDictionaryAnyObject	242895	239324	-1.5%	1.01x(?)
ObjectiveCBridgeToNSSet	18756	18586	-0.9%	1.01x(?)
TwoSum	9886	9831	-0.6%	1.01x(?)
ArrayValueProp2	20116	19857	-1.3%	1.01x
ArraySubscript	4813	4820	+0.1%	1.00x(?)
ObjectiveCBridgeToNSString	1101	1101	+0.0%	1.00x
DictionarySwapOfObjects	26624	26649	+0.1%	1.00x(?)
StackPromo	85639	85452	-0.2%	1.00x(?)
PopFrontArray	13370	13369	-0.0%	1.00x(?)
RecursiveOwnedParameter	10974	10981	+0.1%	1.00x(?)
ObjectiveCBridgeStubToNSString	1774	1774	+0.0%	1.00x
ClassArrayGetter	1344	1343	-0.1%	1.00x(?)
SortStrings	5926	5924	-0.0%	1.00x(?)
DictionaryBridge	3892	3900	+0.2%	1.00x(?)
ArrayOfGenericPOD	3432	3429	-0.1%	1.00x(?)
StringWithCString	341865	341848	-0.0%	1.00x(?)
ObjectiveCBridgeFromNSStringForced	2652	2656	+0.1%	1.00x(?)
Prims	14749	14732	-0.1%	1.00x(?)
SortLettersInPlace	2420	2427	+0.3%	1.00x
DictionarySwap	11785	11821	+0.3%	1.00x(?)
Dictionary3OfObjects	3427	3419	-0.2%	1.00x(?)
RangeAssignment	16837	16776	-0.4%	1.00x
ByteSwap	9	9	+0.0%	1.00x
SuperChars	779693	780545	+0.1%	1.00x(?)
XorLoop	19691	19694	+0.0%	1.00x
StringInterpolation	14445	14391	-0.4%	1.00x
Integrate	360	359	-0.3%	1.00x
113	5646172	5668274	+0.4%	1.00x
ErrorHandling	7279	7301	+0.3%	1.00x(?)
ObjectiveCBridgeStubFromNSString	1171	1173	+0.2%	1.00x(?)
ProtocolDispatch	5548	5556	+0.1%	1.00x(?)
ObjectiveCBridgeFromNSSetAnyObject	123847	123641	-0.2%	1.00x(?)
AngryPhonebook	3722	3716	-0.2%	1.00x
ProtocolDispatch2	437	437	+0.0%	1.00x
Dictionary3	2780	2782	+0.1%	1.00x(?)
Dictionary2	6502	6516	+0.2%	1.00x(?)
StrComplexWalk	7736	7754	+0.2%	1.00x(?)
SetIntersect_OfObjects	20157	20147	-0.1%	1.00x(?)
Join	1412	1409	-0.2%	1.00x
ArrayOfRef	10893	10894	+0.0%	1.00x(?)
NSError	696	693	-0.4%	1.00x(?)
DictionaryOfObjects	8725	8768	+0.5%	1.00x
PopFrontArrayGeneric	13165	13132	-0.2%	1.00x(?)
PopFrontUnsafePointer	98856	98837	-0.0%	1.00x(?)
CaptureProp	56451	56330	-0.2%	1.00x(?)
PolymorphicCalls	1206	1205	-0.1%	1.00x(?)
Calculator	1047	1046	-0.1%	1.00x(?)
MapReduce	46373	46442	+0.1%	1.00x(?)
DictionaryLiteral	21709	21702	-0.0%	1.00x(?)
Hanoi	24737	24738	+0.0%	1.00x(?)
OpenClose	406	408	+0.5%	1.00x(?)
DictionaryRemoveOfObjects	64353	64272	-0.1%	1.00x(?)
UTF8Decode	41949	41905	-0.1%	1.00x(?)
SetIsSubsetOf	3167	3161	-0.2%	1.00x(?)
Dictionary	5480	5477	-0.1%	1.00x(?)
NopDeinit	58283	58293	+0.0%	1.00x(?)
SetIntersect	20126	20170	+0.2%	1.00x(?)
SetExclusiveOr	52066	51948	-0.2%	1.00x
NSDictionaryCastToSwift	12917	12934	+0.1%	1.00x(?)
RGBHistogramOfObjects	103954	104375	+0.4%	1.00x(?)
StrToInt	18753	18752	-0.0%	1.00x(?)
ArrayInClass	3885	3884	-0.0%	1.00x(?)
ObjectiveCBridgeFromNSString	5116	5122	+0.1%	1.00x(?)
Sim2DArray	14127	14115	-0.1%	1.00x
SetExclusiveOr_OfObjects	71422	71525	+0.1%	1.00x(?)
MonteCarloPi	53379	53330	-0.1%	1.00x
MonteCarloE	100436	100808	+0.4%	1.00x
SetUnion_OfObjects	54632	54546	-0.2%	1.00x(?)
GlobalClass	0	0	+0.0%	1.00x
HashTest	6021	6024	+0.1%	1.00x(?)
SetIsSubsetOf_OfObjects	3032	3029	-0.1%	1.00x(?)
ArrayAppend	2842	2842	+0.0%	1.00x
DictionaryRemove	31906	31902	-0.0%	1.00x(?)
LinkedList	27875	27750	-0.5%	1.00x(?)
NSStringConversion	2689	2697	+0.3%	1.00x(?)
RGBHistogram	52502	52490	-0.0%	1.00x(?)
ObjectiveCBridgeFromNSSetAnyObjectForced	7883	7845	-0.5%	1.00x(?)
ArrayValueProp4	2651	2652	+0.0%	1.00x(?)
ArrayOfPOD	2331	2334	+0.1%	1.00x(?)
SetUnion	37283	37266	-0.1%	1.00x(?)
Chars	58524	58469	-0.1%	1.00x(?)
StringBuilder	8642	8667	+0.3%	1.00x(?)
BitCount	79	79	+0.0%	1.00x
ArrayLiteral	1096	1096	+0.0%	1.00x
SevenBoom	1516	1523	+0.5%	1.00x(?)
ArrayValueProp	2331	2333	+0.1%	1.00x(?)
RC4	9310	9315	+0.1%	1.00x(?)
Memset	19957	19962	+0.0%	1.00x(?)
Dictionary2OfObjects	8322	8342	+0.2%	1.00x(?)
ArrayAppendReserved	2590	2592	+0.1%	1.00x(?)
ArrayValueProp3	2713	2707	-0.2%	1.00x
ObjectiveCBridgeFromNSDictionaryAnyObjectToString	217147	218565	+0.7%	0.99x(?)
Histogram	14642	14729	+0.6%	0.99x(?)
ObjectAllocation	1419	1434	+1.1%	0.99x(?)
Walsh	13013	13134	+0.9%	0.99x(?)
ObjectiveCBridgeToNSArray	30523	30898	+1.2%	0.99x(?)
ObjectiveCBridgeStubToArrayOfNSString	31296	31555	+0.8%	0.99x
ObjectiveCBridgeFromNSArrayAnyObject	70172	70715	+0.8%	0.99x(?)
DeadArray	116994	118701	+1.5%	0.99x
ArrayOfGenericRef	11614	11794	+1.6%	0.98x
ObjectiveCBridgeToNSDictionary	18634	19077	+2.4%	0.98x(?)
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	123062	125261	+1.8%	0.98x(?)
ObjectiveCBridgeStubFromArrayOfNSString	111455	113527	+1.9%	0.98x(?)
StaticArray	17663	18383	+4.1%	0.96x

**Hardware Overview** Model Name: Mac mini Model Identifier: Macmini7,1 Processor Name: Intel Core i5 Processor Speed: 2.8 GHz Number of Processors: 1 Total Number of Cores: 2 L2 Cache (per Core): 256 KB L3 Cache: 3 MB Memory: 16 GB

[Compare Perf] Add support for markdown and html

shahmishal changed the title ~~[Compare Perf] Add support for markdown and html~~ [WIP][Compare Perf] Add support for markdown and html Mar 15, 2016

[Compare Perf] Add support for markdown and html

cb23837

shahmishal added a commit that referenced this pull request Mar 22, 2016

Merge pull request #1682 from shahmishal/master

243e359

[Compare Perf] Add support for markdown and html

shahmishal merged commit 243e359 into swiftlang:master Mar 22, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP][Compare Perf] Add support for markdown and html #1682

[WIP][Compare Perf] Add support for markdown and html #1682

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

practicalswift commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

gottesmm commented Mar 14, 2016

Uh oh!

gottesmm commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 15, 2016

Uh oh!

swift-ci commented Mar 15, 2016

Uh oh!

practicalswift commented Mar 15, 2016

Uh oh!

shahmishal commented Mar 22, 2016

Uh oh!

swift-ci commented Mar 22, 2016

Uh oh!

shahmishal commented Mar 22, 2016

Uh oh!

swift-ci commented Mar 22, 2016

Uh oh!

Uh oh!

[WIP][Compare Perf] Add support for markdown and html #1682

[WIP][Compare Perf] Add support for markdown and html #1682

Uh oh!

Conversation

shahmishal commented Mar 14, 2016

What's in this pull request?

Resolved bug number: N/A

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

practicalswift commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

gottesmm commented Mar 14, 2016

Uh oh!

gottesmm commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 14, 2016

Uh oh!

shahmishal commented Mar 15, 2016

Uh oh!

swift-ci commented Mar 15, 2016

Uh oh!

practicalswift commented Mar 15, 2016

Uh oh!

shahmishal commented Mar 22, 2016

Uh oh!

swift-ci commented Mar 22, 2016

Build comment file:

Uh oh!

shahmishal commented Mar 22, 2016

Uh oh!

swift-ci commented Mar 22, 2016

Build comment file:

Uh oh!

Uh oh!