Benchmark cmake: Allow building from a Swift root. #23814

atrick · 2019-04-05T16:45:59Z

And fix the SWIFT_LIBRARY_PATH option so that it also sets the
RPATH.

atrick · 2019-04-05T16:46:14Z

@swift-ci benchmark.

swift-ci · 2019-04-05T17:33:23Z

Build failed before running benchmark.

atrick · 2019-04-05T21:28:54Z

@swift-ci benchmark

swift-ci · 2019-04-05T22:37:47Z

Build failed before running benchmark.

atrick · 2019-04-05T23:50:53Z

@swift-ci benchmark

swift-ci · 2019-04-06T01:05:02Z

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Improvement
ObjectiveCBridgeStubFromNSDateRef	4390	4050	-7.7%	1.08x (?)

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

atrick · 2019-04-06T07:12:37Z

@swift-ci test.

atrick · 2019-04-06T07:14:40Z

@gottesmm I'm done with these cmake fixes. I didn't do any auxiliary cleanup. It should be trivial to review.

swift-ci · 2019-04-06T07:14:46Z

Build failed
Swift Test Linux Platform
Git Sha - ab24cae1b990cddb87a62a608a881aec24d7ccda

swift-ci · 2019-04-06T07:16:57Z

Build failed
Swift Test OS X Platform
Git Sha - ab24cae1b990cddb87a62a608a881aec24d7ccda

gottesmm

Some quick questions. Then I want to look one more time.

gottesmm · 2019-04-07T17:46:09Z

benchmark/CMakeLists.txt

+endif()
+if (SWIFT_RPATH)
+message("--   SWIFT_RPATH        = ${SWIFT_RPATH}")
+message("---  ** WARNING ** Benchmarking against Swift-in-the-OS")


Nice warning!

gottesmm · 2019-04-07T17:49:45Z

benchmark/README.md

+the performance of the Swift standard library installed on the hosts,
+not the one included in the Swift root. This could be built as such:
+
+    cmake .. -G Ninja -DSWIFT_DARWIN_XCRUN_TOOLCHAIN=XcodeDefault -DSWIFT_EXEC=`xcrun --toolchain XcodeDefault --find swiftc`


Can you document SWIFT_DARWIN_XCRUN_TOOLCHAIN above?

I almost wonder if it would make sense to have a separate section for building against Swift-in-the-OS? Your thoughts?

gottesmm · 2019-04-07T17:50:18Z

benchmark/cmake/modules/AddSwiftBenchmarkSuite.cmake

-    set(SWIFT_LIBRARY_PATH "${tmp_dir}/lib/swift")
+  # Check if SWIFT_EXEC is from a build vs. install path.
+  set(SWIFT_EXEC_INSTALLED FALSE)
+  if(SWIFT_BENCHMARK_BUILT_STANDALONE AND NOT EXISTS "${SWIFT_EXEC}../lib/swift/${BENCH_COMPILE_ARCHOPTS_PLATFORM}/Swift.swiftmodule")


Is this correct? Shouldn't SWIFT_EXEC have a / after? Also, TBH, why shouldn't SWIFT_BENCHMARK_BUILT_STANDALONE just imply this?

atrick · 2019-04-09T01:54:01Z

@gottesmm Getting all the right behavior without adding a new option was too tricky. So now we have
SWIFT_BENCHMARK_USE_OS_LIBRARIES, and no weird inference during configuration.

There's nothing we can really infer from the SWIFT_BENCHMARK_BUILT_STANDALONE flag, except that SWIFT_LIBRARY_PATH was not already set by the LLVM config.

atrick · 2019-04-09T01:54:09Z

@swift-ci benchmark

swift-ci · 2019-04-09T02:44:43Z

No performance and code size changes

atrick · 2019-04-09T21:51:54Z

benchmark/CMakeLists.txt

 message("--   SWIFT_LIBRARY_PATH = ${SWIFT_LIBRARY_PATH}")
-message("--   CLANG_EXEC = ${CLANG_EXEC}")
+if (SWIFT_RPATH_BASE)


Add a top-level comment explaining what an rpath base is

Cleanup and document the configuration of the library path and rpath. With SWIFT_BENCHMARK_USE_OS_LIBRARIES, it's now possible to directly build benchmarks for a target device and run those benchmarks on the device without building or installing Swift. It's also possible now to specify an absolute SWIFT_LIBRARY_PATH to be used as an rpath so installation can be skipped.

atrick · 2019-04-10T16:32:06Z

@swift-ci benchmark

atrick · 2019-04-10T16:32:15Z

@swift-ci test

swift-ci · 2019-04-10T16:34:21Z

Build failed
Swift Test Linux Platform
Git Sha - c2b9f84638876398ffc2b344857a908333960528

atrick · 2019-04-10T16:37:26Z

@swift-ci benchmark

atrick · 2019-04-10T16:39:04Z

@swift-ci benchmark benchmark benchmark

atrick · 2019-04-10T16:40:00Z

@swift-ci test

atrick · 2019-04-10T17:38:22Z

@swift-ci benchmark

atrick · 2019-04-10T18:05:04Z

@swift-ci test OSX platform

swift-ci · 2019-04-10T18:39:03Z

Build failed
Swift Test OS X Platform
Git Sha - e622103c86110efa6bb1985ade947d22ba4943d2

atrick · 2019-04-10T18:39:34Z

@swift-ci test OS X

swift-ci · 2019-04-10T19:30:15Z

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
DataAppendDataLargeToLarge	36600	50400	+37.7%	0.73x (?)
ObjectiveCBridgeStubFromNSStringRef	167	184	+10.2%	0.91x (?)
Improvement
ObjectiveCBridgeStubToNSDate2	1400	1220	-12.9%	1.15x (?)

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

atrick requested a review from gottesmm April 5, 2019 16:46

gottesmm reviewed Apr 7, 2019

View reviewed changes

atrick commented Apr 9, 2019

View reviewed changes

atrick merged commit 43f9df2 into swiftlang:master Apr 10, 2019

atrick deleted the benchmark-cmake branch May 8, 2019 21:35

Benchmark cmake: Allow building from a Swift root. #23814

Benchmark cmake: Allow building from a Swift root. #23814

Uh oh!

Conversation

atrick commented Apr 5, 2019

Uh oh!

atrick commented Apr 5, 2019

Uh oh!

swift-ci commented Apr 5, 2019

Uh oh!

atrick commented Apr 5, 2019

Uh oh!

swift-ci commented Apr 5, 2019

Uh oh!

atrick commented Apr 5, 2019

Uh oh!

swift-ci commented Apr 6, 2019

Performance: -Osize

Uh oh!

atrick commented Apr 6, 2019

Uh oh!

atrick commented Apr 6, 2019

Uh oh!

swift-ci commented Apr 6, 2019

Uh oh!

swift-ci commented Apr 6, 2019

Uh oh!

gottesmm left a comment

Choose a reason for hiding this comment

Uh oh!

gottesmm Apr 7, 2019

Choose a reason for hiding this comment

Uh oh!

gottesmm Apr 7, 2019

Choose a reason for hiding this comment

Uh oh!

gottesmm Apr 7, 2019

Choose a reason for hiding this comment

Uh oh!

atrick commented Apr 9, 2019

Uh oh!

atrick commented Apr 9, 2019

Uh oh!

swift-ci commented Apr 9, 2019

No performance and code size changes

Uh oh!

atrick Apr 9, 2019

Choose a reason for hiding this comment

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

swift-ci commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

swift-ci commented Apr 10, 2019

Uh oh!

atrick commented Apr 10, 2019

Uh oh!

swift-ci commented Apr 10, 2019

Performance: -O

Uh oh!

Uh oh!