Skip to content

Commit 283713d

Browse files
davezarzyckidavezarzycki
committed
[Testing] Formalize stress tests
Stress tests are, by definition, stressful. They intentionally burn a lot of resources by using randomness to hopefully surface state machine bugs. Additionally, many stress tests are multi-threaded these days and they may attempt to use all of the available CPUs to better uncover bugs. In isolation, this is not a problem, but the test suite as a whole assumes that individual tests are single threaded and therefore running multiple stress tests at once can quickly spiral out of control. This change formalizes stress tests and then treats them like long tests, i.e. tested via 'check-swift-all' and otherwise opt-in. Finally, with this change, the CI build bots might need to change if they are still only testing 'validation' instead of all of the tests. I see three options: 1) Run all of the tests. -- There are very few long tests left these days, and the additional costs seems small relative to the cost of the whole validation test suite before this change. 2) Continue checking 'validation', now sans stress tests. 3) Check 'validation', *then* the stress tests. If the former doesn't pass, then there is no point in the latter, and by running the stress tests separately, they stand a better chance of uncovering bugs and not overwhelming build bot resources.
1 parent dbfee4d commit 283713d

19 files changed

+30
-2
lines changed

‎docs/Testing.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ The testsuite is split into four subsets:
3333
* Validation testsuite, located under ``swift/validation-test``.
3434
* Unit tests, located under ``swift/unittests``.
3535
* Long tests, which are marked with ``REQUIRES: long_test``.
36+
* Stress tests, which are marked with ``REQUIRES: stress_test``.
3637

3738
### Running the LLVM lit-based testsuite
3839

@@ -126,8 +127,9 @@ Besides ``check-swift``, other targets are also available. Here's the full list:
126127

127128
* ``check-swift``: Runs tests from the ``${SWIFT_SOURCE_ROOT}/test`` directory.
128129
* ``check-swift-only_validation``: Runs tests from the ``${SWIFT_SOURCE_ROOT}/validation-test`` directory.
129-
* ``check-swift-validation``: Runs the primary and validation tests, without the long tests.
130+
* ``check-swift-validation``: Runs the primary and validation tests, without the long tests or stress tests.
130131
* ``check-swift-only_long``: Runs long tests only.
132+
* ``check-swift-only_stress``: Runs stress tests only.
131133
* ``check-swift-all``: Runs all tests (primary, validation, and long).
132134
* ``SwiftUnitTests``: Builds all unit tests. Executables are located under
133135
``${SWIFT_BUILD_DIR}/unittests`` and must be run individually.

‎test/CMakeLists.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,7 @@ set(TEST_SUBSETS
146146
all
147147
only_validation
148148
only_long
149+
only_stress
149150
)
150151

151152
if(NOT "${COVERAGE_DB}" STREQUAL "")
@@ -271,12 +272,14 @@ foreach(SDK ${SWIFT_SDKS})
271272
if((test_subset STREQUAL "primary") OR
272273
(test_subset STREQUAL "validation") OR
273274
(test_subset STREQUAL "only_long") OR
275+
(test_subset STREQUAL "only_stress") OR
274276
(test_subset STREQUAL "all"))
275277
list(APPEND directories "${test_bin_dir}")
276278
endif()
277279
if((test_subset STREQUAL "validation") OR
278280
(test_subset STREQUAL "only_validation") OR
279281
(test_subset STREQUAL "only_long") OR
282+
(test_subset STREQUAL "only_stress") OR
280283
(test_subset STREQUAL "all"))
281284
list(APPEND directories "${validation_test_bin_dir}")
282285
list(APPEND dependencies ${validation_test_dependencies})

‎test/Runtime/weak-reference-racetests-dispatch.swift

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// RUN: %target-run-simple-swift
22
// REQUIRES: executable_test
3-
3+
// REQUIRES: stress_test
44
// REQUIRES: objc_interop
55

66
import StdlibUnittest

‎test/Runtime/weak-reference-racetests.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
// RUN: %target-run-simple-swift
22
// REQUIRES: executable_test
3+
// REQUIRES: stress_test
34
// UNSUPPORTED: nonatomic_rc
45

56
import StdlibUnittest

‎test/SILOptimizer/string_switch.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
// RUN: %target-build-swift -O %s -module-name=test -Xllvm -sil-disable-pass=FunctionSignatureOpts -emit-sil | %FileCheck %s
33
// RUN: %target-run %t.out
44
// REQUIRES: swift_stdlib_no_asserts,optimized_stdlib
5+
// REQUIRES: stress_test
56
// UNSUPPORTED: nonatomic_rc
67

78
import StdlibUnittest

‎test/Sanitizers/tsan-ignores-arc-locks.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// RUN: %target-build-swift -target %sanitizers-target-triple -sanitize=thread %s -o %t_binary
22
// RUN: %env-TSAN_OPTIONS=ignore_interceptors_accesses=1:halt_on_error=1 %target-run %t_binary
33
// REQUIRES: executable_test
4+
// REQUIRES: stress_test
45
// REQUIRES: tsan_runtime
56
// REQUIRES: objc_interop
67

‎test/Sanitizers/tsan-inout.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
// RUN: not env %env-TSAN_OPTIONS=abort_on_error=0 %target-run %t/tsan-binary 2>&1 | %FileCheck %s
77
// RUN: not env %env-TSAN_OPTIONS=abort_on_error=0:ignore_interceptors_accesses=0 %target-run %t/tsan-binary 2>&1 | %FileCheck %s --check-prefix CHECK-INTERCEPTORS-ACCESSES
88
// REQUIRES: executable_test
9+
// REQUIRES: stress_test
910
// REQUIRES: objc_interop
1011
// REQUIRES: tsan_runtime
1112

‎test/Sanitizers/tsan-type-metadata.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// RUN: %target-swiftc_driver -target %sanitizers-target-triple -sanitize=thread %s -o %t_binary
22
// RUN: %env-TSAN_OPTIONS=ignore_interceptors_accesses=1:halt_on_error=1 %target-run %t_binary
33
// REQUIRES: executable_test
4+
// REQUIRES: stress_test
45
// REQUIRES: objc_interop
56
// REQUIRES: tsan_runtime
67

‎test/Sanitizers/witness_table_lookup.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// RUN: %target-build-swift -sanitize=thread -target %sanitizers-target-triple %s -o %t_binary
22
// RUN: %env-TSAN_OPTIONS=ignore_interceptors_accesses=1:halt_on_error=1 %target-run %t_binary
33
// REQUIRES: executable_test
4+
// REQUIRES: stress_test
45
// REQUIRES: tsan_runtime
56

67
// https://bugs.swift.org/browse/SR-6622

‎test/Unit/lit.cfg

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,10 @@ elif swift_test_subset == 'only_long':
5858
# FIXME: this doesn't exclude not-long tests from the only_long subset.
5959
# Currently those tests are very fast so it doesn't matter much.
6060
pass
61+
elif swift_test_subset == 'only_stress':
62+
# FIXME: this doesn't exclude not-stress tests from the only_stress subset.
63+
# Currently those tests are very fast so it doesn't matter much.
64+
pass
6165
else:
6266
lit_config.fatal("Unknown test mode %r" % swift_test_subset)
6367

‎test/lit.cfg

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -504,10 +504,15 @@ if swift_test_subset in ['primary', 'validation', 'only_validation']:
504504
pass
505505
elif swift_test_subset == 'all':
506506
config.available_features.add("long_test")
507+
config.available_features.add("stress_test")
507508
elif swift_test_subset == 'only_long':
508509
config.available_features.add("long_test")
509510
config.limit_to_features.add("long_test")
510511
config.limit_to_features.discard("executable_test")
512+
elif swift_test_subset == 'only_stress':
513+
config.available_features.add("stress_test")
514+
config.limit_to_features.add("stress_test")
515+
config.limit_to_features.discard("executable_test")
511516
else:
512517
lit_config.fatal("Unknown test mode %r" % swift_test_subset)
513518

‎test/stdlib/CommandLine.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
// RUN: %target-build-swift %t/CommandLineStressTest.o %t/CommandLineStressTestSwift.o -o %t/CommandLineStressTest
77
// RUN: %target-run %t/CommandLineStressTest foo bar baz qux quux corge grault garply waldo fred plugh xyzzy and thud
88
// REQUIRES: executable_test
9+
// REQUIRES: stress_test
910
// UNSUPPORTED: nonatomic_rc
1011

1112
// This file is an empty stub to call into the command line stress test which

‎utils/run-test

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ TEST_SUBSETS = [
4242
'all',
4343
'only_validation',
4444
'only_long',
45+
'only_stress',
4546
]
4647

4748
SWIFT_SOURCE_DIR = os.path.join(SWIFT_SOURCE_ROOT, 'swift')

‎validation-test/StdlibUnittest/RaceTest.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
// RUN: %target-build-swift -Xfrontend -disable-access-control -module-name a %s -o %t.out
22
// RUN: %target-run %t.out | %FileCheck %s
3+
// REQUIRES: stress_test
34
// UNSUPPORTED: nonatomic_rc
45

56
import StdlibUnittest

‎validation-test/stdlib/ArrayBridging.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
// REQUIRES: objc_interop
1010
// REQUIRES: executable_test
11+
// REQUIRES: stress_test
1112
// UNSUPPORTED: nonatomic_rc
1213

1314
import StdlibUnittest

‎validation-test/stdlib/AtomicInt.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
// RUN: %target-build-swift -module-name a %s -o %t.out -O
44
// RUN: %target-run %t.out
55
// REQUIRES: executable_test
6+
// REQUIRES: stress_test
67
// UNSUPPORTED: nonatomic_rc
78

89
import SwiftPrivate

‎validation-test/stdlib/DictionaryBridging.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
// RUN: %target-build-swift -Xfrontend -disable-access-control -I %S/Inputs/SlurpFastEnumeration/ %t/main.swift %S/Inputs/DictionaryKeyValueTypes.swift %S/Inputs/DictionaryKeyValueTypesObjC.swift -Xlinker %t/SlurpFastEnumeration.o -o %t.out -O
66
// RUN: %target-run %t.out
77
// REQUIRES: executable_test
8+
// REQUIRES: stress_test
89

910
// REQUIRES: objc_interop
1011
// UNSUPPORTED: nonatomic_rc

‎validation-test/stdlib/ErrorProtocol.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
// RUN: %target-run-simple-swift
22
// REQUIRES: executable_test
3+
// REQUIRES: stress_test
34
// REQUIRES: objc_interop
45
// UNSUPPORTED: nonatomic_rc
56

‎validation-test/stdlib/StringSlicesConcurrentAppend.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
// RUN: %target-run-simple-swift
22
// REQUIRES: executable_test
3+
// REQUIRES: stress_test
34

45
import StdlibUnittest
56
import SwiftPrivatePthreadExtras

0 commit comments

Comments
 (0)