Add simple benchmarks for JSON coding #20886

Catfish-Man · 2018-11-29T21:56:07Z

This is primitive, but it'll give us a baseline to work from

Catfish-Man · 2018-11-29T21:56:51Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-29T21:56:59Z

@swift-ci please smoke test

benchmark/single-source/JSON.swift

swift-ci · 2018-11-29T22:47:11Z

Build comment file:

Performance: -O

TEST	MIN	MAX	MEAN	MAX_RSS
Added
JSONPerfDecode	1115	2329	1528	—
JSONPerfEncode	645	652	648	—

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataCountSmall	37	34	-8.1%	1.09x
DataCountMedium	40	37	-7.5%	1.08x (?)
Added
JSONPerfDecode	1200	1977	1472	—
JSONPerfEncode	699	707	702	—

Performance: -Onone

TEST	MIN	MAX	MEAN	MAX_RSS
Added
JSONPerfDecode	1220	1932	1466	—
JSONPerfEncode	695	730	717	—

✅	Benchmark Check Report
⚠️⏱	`JSONPerfDecode` execution took at least 1062 μs. _{Decrease the workload of JSONPerfDecode by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`JSONPerfDecode` has very wide range of memory used between independent, repeated measurements. _{JSONPerfDecode mem_pages [i1, i2]: min=[156, 157] 𝚫=1 R=[11, 20]}
⚠️Ⓜ️	`JSONPerfEncode` has very wide range of memory used between independent, repeated measurements. _{JSONPerfEncode mem_pages [i1, i2]: min=[100, 100] 𝚫=0 R=[8, 16]}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

--------------

itaiferber

Looks like a reasonable start to me. For evolving this in the future, though: what do we directly expect to measure here? Since JSONEncoder/JSONDecoder rely on JSONSerialization, we'll also be picking up any fluctuations there over time; is that acceptable for these benchmarks? Are the tests here intending to measure the cost of the JSON side of things, the Codable side of things, or both?

(I ask the latter because I've long wanted to introduce a base Encoder/Decoder that does nothing but collect an object graph into Swift collections without encoding to any format — ideally allowing us to unit-test/benchmark the Codable side of things instead of the underlying format.)

itaiferber · 2018-11-29T22:46:07Z

benchmark/single-source/JSON.swift

+  var poems = Array(repeating: Jabberwocky(), count: 6)
+  var encoder: JSONEncoder = JSONEncoder()
+  var decoder: JSONDecoder = JSONDecoder()
+  var data: Data! = nil


Is there a reason to give this a default value?

Probably not

Catfish-Man · 2018-11-29T23:04:19Z

@itaiferber I wanted an end-to-end test so that we can catch regressions on the OS side as well (and just because that's what users actually care about). I agree it would be good to have tests for the individual pieces as well.

Catfish-Man · 2018-11-29T23:10:13Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-29T23:14:14Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-29T23:18:39Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-29T23:18:57Z

@swift-ci please smoke test

Catfish-Man · 2018-11-29T23:20:20Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-29T23:24:40Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-30T00:18:53Z

@swift-ci please smoke benchmark

Catfish-Man · 2018-11-30T00:19:03Z

@swift-ci please smoke test

benchmark/single-source/JSON.swift

lancep · 2018-11-30T00:29:24Z

@Catfish-Man I think you want to do just a plain benchmark instead of smoke benchmark

Catfish-Man · 2018-11-30T00:40:14Z

@swift-ci please benchmark

swift-ci · 2018-11-30T01:15:56Z

Build comment file:

Performance: -O

TEST	MIN	MAX	MEAN	MAX_RSS
Added
JSONPerfDecode	598	599	599	—
JSONPerfEncode	651	658	654	—

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataCountSmall	37	34	-8.1%	1.09x (?)
DataCountMedium	40	37	-7.5%	1.08x (?)
Added
JSONPerfDecode	587	594	589	—
JSONPerfEncode	680	688	683	—

Performance: -Onone

TEST	MIN	MAX	MEAN	MAX_RSS
Added
JSONPerfDecode	627	634	630	—
JSONPerfEncode	716	724	720	—

✅	Benchmark Check Report
⚠️Ⓜ️	`JSONPerfDecode` has very wide range of memory used between independent, repeated measurements. _{JSONPerfDecode mem_pages [i1, i2]: min=[169, 168] 𝚫=1 R=[18, 1]}
⚠️Ⓜ️	`JSONPerfEncode` has very wide range of memory used between independent, repeated measurements. _{JSONPerfEncode mem_pages [i1, i2]: min=[161, 161] 𝚫=0 R=[24, 16]}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

--------------

Catfish-Man · 2018-11-30T01:17:55Z

Those numbers look MUCH more stable, excellent

swift-ci · 2018-11-30T01:30:46Z

Build comment file:

Performance: -O

TEST	MIN	MAX	MEAN	MAX_RSS
Added
JSONPerfDecode	557	564	560	—
JSONPerfEncode	682	690	685	—

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataCountSmall	37	34	-8.1%	1.09x (?)
DataCountMedium	40	37	-7.5%	1.08x (?)
Added
JSONPerfDecode	578	590	584	—
JSONPerfEncode	694	705	700	—

Performance: -Onone

TEST	MIN	MAX	MEAN	MAX_RSS
Added
JSONPerfDecode	637	665	648	—
JSONPerfEncode	703	711	706	—

✅	Benchmark Check Report
⚠️Ⓜ️	`JSONPerfDecode` has very wide range of memory used between independent, repeated measurements. _{JSONPerfDecode mem_pages [i1, i2]: min=[155, 155] 𝚫=0 R=[27, 18]}
⚠️Ⓜ️	`JSONPerfEncode` has very wide range of memory used between independent, repeated measurements. _{JSONPerfEncode mem_pages [i1, i2]: min=[156, 156] 𝚫=0 R=[8, 16]}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

--------------

Catfish-Man · 2018-11-30T01:38:23Z

@swift-ci please test and merge

Catfish-Man · 2018-11-30T01:44:12Z

@swift-ci please test and merge

Catfish-Man · 2018-11-30T01:46:33Z

@swift-ci please test and merge

Catfish-Man requested a review from moiseev November 29, 2018 21:56

Catfish-Man self-assigned this Nov 29, 2018

Catfish-Man requested a review from itaiferber November 29, 2018 21:58

lancep reviewed Nov 29, 2018

View reviewed changes

benchmark/single-source/JSON.swift Outdated Show resolved Hide resolved

lancep reviewed Nov 29, 2018

View reviewed changes

benchmark/single-source/JSON.swift Outdated Show resolved Hide resolved

itaiferber reviewed Nov 29, 2018

View reviewed changes

Catfish-Man force-pushed the someone-called-json-is-decoding-things branch 2 times, most recently from fbdceb6 to 55e74f9 Compare November 29, 2018 23:09

Catfish-Man force-pushed the someone-called-json-is-decoding-things branch from 55e74f9 to 4b83778 Compare November 29, 2018 23:44

moiseev reviewed Nov 30, 2018

View reviewed changes

benchmark/single-source/JSON.swift Outdated Show resolved Hide resolved

Catfish-Man force-pushed the someone-called-json-is-decoding-things branch from 4b83778 to 805f82e Compare November 30, 2018 01:37

Add simple benchmarks for JSON coding

c129066

Catfish-Man force-pushed the someone-called-json-is-decoding-things branch from 260c9aa to c129066 Compare November 30, 2018 01:42

swift-ci merged commit ece1bfc into swiftlang:master Nov 30, 2018

Catfish-Man deleted the someone-called-json-is-decoding-things branch December 1, 2018 00:51

Add simple benchmarks for JSON coding #20886

Add simple benchmarks for JSON coding #20886

Uh oh!

Conversation

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Uh oh!

Uh oh!

swift-ci commented Nov 29, 2018

Build comment file:

Performance: -O

Performance: -Osize

Performance: -Onone

Uh oh!

itaiferber left a comment

Choose a reason for hiding this comment

Uh oh!

itaiferber Nov 29, 2018

Choose a reason for hiding this comment

Uh oh!

Catfish-Man Nov 29, 2018

Choose a reason for hiding this comment

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 29, 2018

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

Uh oh!

lancep commented Nov 30, 2018

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

swift-ci commented Nov 30, 2018

Build comment file:

Performance: -O

Performance: -Osize

Performance: -Onone

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

swift-ci commented Nov 30, 2018

Build comment file:

Performance: -O

Performance: -Osize

Performance: -Onone

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

Catfish-Man commented Nov 30, 2018

Uh oh!

Uh oh!