1
- # Swift Benchmark Suite
1
+ Swift Benchmark Suite
2
+ =====================
2
3
3
4
This directory contains the Swift Benchmark Suite.
4
5
5
- ## Running Swift Benchmarks
6
+ Running Swift Benchmarks
7
+ ------------------------
6
8
7
9
To run Swift benchmarks, pass the ` --benchmark ` flag to ` build-script ` . The
8
10
current benchmark results will be compared to the previous run's results if
@@ -14,17 +16,8 @@ impacting changes, and run the benchmarks again. Upon benchmark completion, the
14
16
benchmark results for the development branch will be compared to the most
15
17
recent benchmark results for ` master ` .
16
18
17
- ## Building the Swift Benchmarks
18
-
19
- The swift benchmark suite currently supports building with CMake and
20
- SwiftPM. We support the following platforms respectively.
21
-
22
- * CMake: macOS, iOS, tvOS, watchOS
23
- * SwiftPM: macOS, linux
24
-
25
- We describe how to build both standalone and with build-script below.
26
-
27
- ### build-script invoking CMake
19
+ Building with build-script
20
+ --------------------------
28
21
29
22
By default, Swift benchmarks for OS X are compiled during the Swift build
30
23
process. To build Swift benchmarks for additional platforms, pass the following
@@ -40,7 +33,8 @@ drivers dynamically link Swift standard library dylibs from a path
40
33
relative to their run-time location (../lib/swift) so the standard
41
34
library should be distributed alongside them.
42
35
43
- ### CMake Standalone (no build-script)
36
+ Building Independently
37
+ ----------------------
44
38
45
39
To build the Swift benchmarks using only an Xcode installation: install an
46
40
Xcode version with Swift support, install cmake 2.8.12, and ensure Xcode is
@@ -116,57 +110,8 @@ installed libraries instead, enable
116
110
This will reflect the performance of the Swift standard library
117
111
installed on the device, not the one included in the Swift root.
118
112
119
- ### build-script using SwiftPM+LLBuild
120
-
121
- To build the benchmarks using build-script/swiftpm, one must build both
122
- swiftpm/llbuild as part of one's build and create a "just-built" toolchain. This
123
- toolchain is then used by build-script to compile the benchmarks. This is
124
- accomplished by passing to build-script the following options:
125
-
126
- ```
127
- swift-source$ swift/utils/build-script --swiftpm --llbuild --install-swift --install-swiftpm --install-llbuild --toolchain-benchmarks
128
- ```
129
-
130
- build-script will then compile the toolchain and then build the benchmarks 3
131
- times, once for each optimization level, at the path
132
- ` ./build/benchmarks-$PLATFORM-$ARCH/bin/Benchmark_$OPT ` :
133
-
134
- ### Standalone SwiftPM/LLBuild
135
-
136
- The benchmark suite can be built with swiftpm/llbuild without needing any help
137
- from build-script by invoking swift build in the benchmark directory:
138
-
139
- ```
140
- swift-source/swift/benchmark$ swift build -configuration release
141
- swift-source/swift/benchmark$ .build/release/SwiftBench
142
- #,TEST,SAMPLES,MIN(μs),MAX(μs),MEAN(μs),SD(μs),MEDIAN(μs)
143
- 1,Ackermann,1,169,169,169,0,169
144
- 2,AngryPhonebook,1,2044,2044,2044,0,2044
145
- ...
146
- ```
147
-
148
- ## Editing in Xcode
149
-
150
- It is now possible to work on swiftpm benchmarks in Xcode! This is done by using
151
- the ability swiftpm build of the benchmarks to generate an xcodeproject. This is
152
- done by running the commands:
153
-
154
- ```
155
- swift-source/swift/benchmark$ swift package generate-xcodeproj
156
- generated: ./swiftbench.xcodeproj
157
- swift-source/swift/benchmark$ open swiftbench.xcodeproj
158
- ```
159
-
160
- Assuming that Xcode is installed on ones system, this will open the project in
161
- Xcode. The benchmark binary is built by the target 'SwiftBench'.
162
-
163
- ** NOTE: Files added to the Xcode project will not be persisted back to the
164
- package! To add new benchmarks follow the instructions from the section below!**
165
-
166
- ** NOTE: By default if one just builds/runs the benchmarks in Xcode, the
167
- benchmarks will be compiled with -Onone!**
168
-
169
- ## Using the Benchmark Driver
113
+ Using the Benchmark Driver
114
+ --------------------------
170
115
171
116
### Usage
172
117
@@ -209,39 +154,32 @@ You can use test numbers instead of test names like this:
209
154
Test numbers are not stable in the long run, adding and removing tests from the
210
155
benchmark suite will reorder them, but they are stable for a given build.
211
156
212
- ## Using the Harness Generator
157
+ Using the Harness Generator
158
+ ---------------------------
213
159
214
160
` scripts/generate_harness/generate_harness.py ` runs ` gyb ` to automate generation
215
161
of some benchmarks.
216
162
217
163
** FIXME ** ` gyb ` should be invoked automatically during the
218
164
build so that manually invoking ` generate_harness.py ` is not required.
219
165
220
- ## Adding New Benchmarks
166
+ Adding New Benchmarks
167
+ ---------------------
221
168
222
- Adding a new benchmark requires some boilerplate updates. To ease this (and
223
- document the behavior), a harness generator script is provided for both
224
- single/multiple file tests.
169
+ The harness generator supports both single and multiple file tests.
225
170
226
- To add a new single file test, execute the following script with the new of the
227
- benchmark:
171
+ To add a new single file test, execute the following script with the new of the benchmark:
228
172
229
173
```
230
174
swift-source$ ./swift/benchmark/scripts/create_benchmark.py YourTestNameHere
231
175
```
232
176
233
177
The script will automatically:
234
-
235
178
1 . Add a new Swift file (` YourTestNameHere.swift ` ), built according to
236
179
the template below, to the ` single-source ` directory.
237
180
2 . Add the filename of the new Swift file to ` CMakeLists.txt ` .
238
181
3 . Edit ` main.swift ` by importing and registering your new Swift module.
239
182
240
- No changes are needed to the Package.swift file since the benchmark's
241
- Package.swift is set to dynamically lookup each Swift file in ` single-source `
242
- and translate each of those individual .swift files into individual modules. So
243
- the new test file will be automatically found.
244
-
245
183
To add a new multiple file test:
246
184
247
185
1 . Add a new directory and files under the ` multi-source ` directory as
@@ -262,9 +200,6 @@ To add a new multiple file test:
262
200
263
201
3 . Edit ` main.swift ` . Import and register your new Swift module.
264
202
265
- No changes are needed to the swiftpm build since it knows how to infer
266
- multi-source libraries automatically from the library structure.
267
-
268
203
** Note:**
269
204
270
205
The benchmark harness will execute the routine referenced by
@@ -309,8 +244,8 @@ public func run_YourTestName(N: Int) {
309
244
The current set of tags are defined by the ` BenchmarkCategory ` enum in
310
245
` TestsUtils.swift ` .
311
246
312
- ## Testing the Benchmark Drivers
313
-
247
+ Testing the Benchmark Drivers
248
+ -----------------------------
314
249
When working on tests, after the initial build
315
250
````
316
251
swift-source$ ./swift/utils/build-script -R -B
0 commit comments