Skip to content

Commit ca48aae

Browse files
committed
[doc] Initial validation-test documentation. It is not complete, but it is a good start.
1 parent 33d77ef commit ca48aae

File tree

1 file changed

+29
-0
lines changed

1 file changed

+29
-0
lines changed

docs/ContinuousIntegration.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,33 @@ A smoke test on Linux does the following:
6767
OS X platform | @swift-ci Please test OS X platform
6868
Linux platform | @swift-ci Please test Linux platform
6969

70+
The core principles of validation testing is that:
71+
72+
1. A validation test should build and run tests for /all/ platforms and all
73+
architectures supported by the CI.
74+
2. A validation test should not be incremental. We want there to be a
75+
definitiveness to a validation test. If one uses a validation test, one
76+
should be sure that there is no nook or cranny in the code base that has not
77+
been tested.
78+
79+
With that being said, a validation test on macOS does the following:
80+
81+
1. Removes the workspace.
82+
2. Builds the compiler.
83+
3. Builds the standard library for macOS and the simulators for all platforms.
84+
4. lldb is /not/ build/tested [[2]](#footnote-2)
85+
5. The tests, validation-tests are run for all simulators and macOS both with
86+
and without optimizations enabled.
87+
88+
A validation test on Linux does the following:
89+
90+
1. Removes the workspace.
91+
2. Builds the compiler.
92+
3. Builds the standard library.
93+
4. lldb is built.
94+
5. Run the swift test and validation-test targets with and without optimization.
95+
6. lldb is tested.
96+
7097
### Benchmarking
7198

7299
Platform | Comment
@@ -110,3 +137,5 @@ FIXME: FILL ME IN!
110137
1. A full build break can prevent other developers from testing their work.
111138
2. A test break can make it difficult for developers to know whether or not their specific commit has broken a test, requiring them to perform an initial clean build, wasting time.
112139
3. @swift-ci pull request testing becomes less effective since one can not perform a test and merge and one must reason about the source of a given failure.
140+
141+
<a name="footnote-2">[2]</a> This is due to unrelated issues relating to running lldb tests on macOS.

0 commit comments

Comments
 (0)