1
- Swift Benchmark Suite
2
- =====================
1
+ # Swift Benchmark Suite
3
2
4
3
This directory contains the Swift Benchmark Suite.
5
4
6
- Running Swift Benchmarks
7
- ------------------------
5
+ ## Running Swift Benchmarks
8
6
9
7
To run Swift benchmarks, pass the ` --benchmark ` flag to ` build-script ` . The
10
8
current benchmark results will be compared to the previous run's results if
@@ -16,8 +14,17 @@ impacting changes, and run the benchmarks again. Upon benchmark completion, the
16
14
benchmark results for the development branch will be compared to the most
17
15
recent benchmark results for ` master ` .
18
16
19
- Building with build-script
20
- --------------------------
17
+ ## Building the Swift Benchmarks
18
+
19
+ The swift benchmark suite currently supports building with CMake and
20
+ SwiftPM. We support the following platforms respectively.
21
+
22
+ * CMake: macOS, iOS, tvOS, watchOS
23
+ * SwiftPM: macOS, linux
24
+
25
+ We describe how to build both standalone and with build-script below.
26
+
27
+ ### build-script invoking CMake
21
28
22
29
By default, Swift benchmarks for OS X are compiled during the Swift build
23
30
process. To build Swift benchmarks for additional platforms, pass the following
@@ -33,8 +40,7 @@ drivers dynamically link Swift standard library dylibs from a path
33
40
relative to their run-time location (../lib/swift) so the standard
34
41
library should be distributed alongside them.
35
42
36
- Building Independently
37
- ----------------------
43
+ ### CMake Standalone (no build-script)
38
44
39
45
To build the Swift benchmarks using only an Xcode installation: install an
40
46
Xcode version with Swift support, install cmake 2.8.12, and ensure Xcode is
@@ -110,8 +116,57 @@ installed libraries instead, enable
110
116
This will reflect the performance of the Swift standard library
111
117
installed on the device, not the one included in the Swift root.
112
118
113
- Using the Benchmark Driver
114
- --------------------------
119
+ ### build-script using SwiftPM+LLBuild
120
+
121
+ To build the benchmarks using build-script/swiftpm, one must build both
122
+ swiftpm/llbuild as part of one's build and create a "just-built" toolchain. This
123
+ toolchain is then used by build-script to compile the benchmarks. This is
124
+ accomplished by passing to build-script the following options:
125
+
126
+ ```
127
+ swift-source$ swift/utils/build-script --swiftpm --llbuild --install-swift --install-swiftpm --install-llbuild --toolchain-benchmarks
128
+ ```
129
+
130
+ build-script will then compile the toolchain and then build the benchmarks 3
131
+ times, once for each optimization level, at the path
132
+ ` ./build/benchmarks-$PLATFORM-$ARCH/bin/Benchmark_$OPT ` :
133
+
134
+ ### Standalone SwiftPM/LLBuild
135
+
136
+ The benchmark suite can be built with swiftpm/llbuild without needing any help
137
+ from build-script by invoking swift build in the benchmark directory:
138
+
139
+ ```
140
+ swift-source/swift/benchmark$ swift build -configuration release
141
+ swift-source/swift/benchmark$ .build/release/SwiftBench
142
+ #,TEST,SAMPLES,MIN(μs),MAX(μs),MEAN(μs),SD(μs),MEDIAN(μs)
143
+ 1,Ackermann,1,169,169,169,0,169
144
+ 2,AngryPhonebook,1,2044,2044,2044,0,2044
145
+ ...
146
+ ```
147
+
148
+ ## Editing in Xcode
149
+
150
+ It is now possible to work on swiftpm benchmarks in Xcode! This is done by using
151
+ the ability swiftpm build of the benchmarks to generate an xcodeproject. This is
152
+ done by running the commands:
153
+
154
+ ```
155
+ swift-source/swift/benchmark$ swift package generate-xcodeproj
156
+ generated: ./swiftbench.xcodeproj
157
+ swift-source/swift/benchmark$ open swiftbench.xcodeproj
158
+ ```
159
+
160
+ Assuming that Xcode is installed on ones system, this will open the project in
161
+ Xcode. The benchmark binary is built by the target 'SwiftBench'.
162
+
163
+ ** NOTE: Files added to the Xcode project will not be persisted back to the
164
+ package! To add new benchmarks follow the instructions from the section below!**
165
+
166
+ ** NOTE: By default if one just builds/runs the benchmarks in Xcode, the
167
+ benchmarks will be compiled with -Onone!**
168
+
169
+ ## Using the Benchmark Driver
115
170
116
171
### Usage
117
172
@@ -154,32 +209,39 @@ You can use test numbers instead of test names like this:
154
209
Test numbers are not stable in the long run, adding and removing tests from the
155
210
benchmark suite will reorder them, but they are stable for a given build.
156
211
157
- Using the Harness Generator
158
- ---------------------------
212
+ ## Using the Harness Generator
159
213
160
214
` scripts/generate_harness/generate_harness.py ` runs ` gyb ` to automate generation
161
215
of some benchmarks.
162
216
163
217
** FIXME ** ` gyb ` should be invoked automatically during the
164
218
build so that manually invoking ` generate_harness.py ` is not required.
165
219
166
- Adding New Benchmarks
167
- ---------------------
220
+ ## Adding New Benchmarks
168
221
169
- The harness generator supports both single and multiple file tests.
222
+ Adding a new benchmark requires some boilerplate updates. To ease this (and
223
+ document the behavior), a harness generator script is provided for both
224
+ single/multiple file tests.
170
225
171
- To add a new single file test, execute the following script with the new of the benchmark:
226
+ To add a new single file test, execute the following script with the new of the
227
+ benchmark:
172
228
173
229
```
174
230
swift-source$ ./swift/benchmark/scripts/create_benchmark.py YourTestNameHere
175
231
```
176
232
177
233
The script will automatically:
234
+
178
235
1 . Add a new Swift file (` YourTestNameHere.swift ` ), built according to
179
236
the template below, to the ` single-source ` directory.
180
237
2 . Add the filename of the new Swift file to ` CMakeLists.txt ` .
181
238
3 . Edit ` main.swift ` by importing and registering your new Swift module.
182
239
240
+ No changes are needed to the Package.swift file since the benchmark's
241
+ Package.swift is set to dynamically lookup each Swift file in ` single-source `
242
+ and translate each of those individual .swift files into individual modules. So
243
+ the new test file will be automatically found.
244
+
183
245
To add a new multiple file test:
184
246
185
247
1 . Add a new directory and files under the ` multi-source ` directory as
@@ -200,6 +262,9 @@ To add a new multiple file test:
200
262
201
263
3 . Edit ` main.swift ` . Import and register your new Swift module.
202
264
265
+ No changes are needed to the swiftpm build since it knows how to infer
266
+ multi-source libraries automatically from the library structure.
267
+
203
268
** Note:**
204
269
205
270
The benchmark harness will execute the routine referenced by
@@ -244,8 +309,8 @@ public func run_YourTestName(N: Int) {
244
309
The current set of tags are defined by the ` BenchmarkCategory ` enum in
245
310
` TestsUtils.swift ` .
246
311
247
- Testing the Benchmark Drivers
248
- -----------------------------
312
+ ## Testing the Benchmark Drivers
313
+
249
314
When working on tests, after the initial build
250
315
````
251
316
swift-source$ ./swift/utils/build-script -R -B
0 commit comments