Add missing docs/Testing.rst

rdar://problem/23359113

Author: bitjammer

docs/Testing.rst

@@ -0,0 +1,365 @@
+:orphan:
+
+.. @raise litre.TestsAreMissing
+
+=============
+Testing Swift
+=============
+
+This document describes how we test the Swift compiler, the Swift runtime, and
+the Swift standard library.
+
+Testing approaches
+==================
+
+We use multiple approaches to test the Swift toolchain.
+
+* LLVM lit-based testsuites for the compiler, runtime and the standard library.
+
+* A selection of open source projects written in Swift.
+
+The LLVM lit-based testsuite
+============================
+
+**Purpose**: primary testsuites for the Swift toolchain.
+
+**Contents**: Functional and regression tests for all toolchain components.
+
+**Run by**:
+
+* Engineers and contributors are expected to run tests from these testsuites
+  locally before commiting.  (Usually on a single platform, and not necessarily
+  all tests.)
+
+* Buildbots run all tests, on all supported platforms.
+
+Running the LLVM lit-based testsuite
+------------------------------------
+
+You can run Swift tests using the ``build-script``, or, alternatively, using
+these targets in the build directory:
+
+* ``check-swift``
+
+  Runs tests from the ``${SWIFT_SOURCE_ROOT}/test`` directory.
+
+* ``check-swift-validation``
+
+  Runs tests from the ``${SWIFT_SOURCE_ROOT}/validation-test`` directory.
+
+* ``check-swift-all``
+
+  Runs all tests.
+
+For day-to-day work on the Swift compiler, using check-swift should be
+sufficient.  The buildbot runs validation tests, so if those are accidentally
+broken, it should not go unnoticed.
+
+Before commiting a large change to a compiler (especially a language change),
+or API changes to the standard library, it is recommended to run validation
+test suite.
+
+For every target above, there are variants for different optimizations:
+
+* the target itself (e.g., ``check-swift``) -- runs execution tests in
+  ``-Onone`` mode;
+
+* the target with ``-optimize`` suffix (e.g., ``check-swift-optimize``) -- runs
+  execution tests in ``-O`` mode; This target will only run tests marked as
+  ``executable_test``.
+
+* the target with ``-optimize-unchecked`` suffix (e.g.,
+  ``check-swift-optimize-unchecked``) -- runs execution tests in
+  ``-Ounchecked`` mode. This target will only run tests marked as
+  ``executable_test``.
+
+If you need to manually run certain tests, you can invoke LLVM's lit.py script
+directly. For example::
+
+    % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv ${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/Parse/
+
+This runs the tests in the test/Parse/ directory targeting the 32-bit iOS
+Simulator. The ``-sv`` options give you a nice progress bar and only show you
+output from the tests that fail.
+
+One downside of using this form is that you're appending relative paths from
+the source directory to the test directory in your build directory. (That is,
+there may not actually be a directory named 'Parse' in
+'test-iphonesimulator-i386/'; the invocation works because there is one in the
+source 'test/' directory.) There is a more verbose form that specifies the
+testing configuration explicitly, which then allows you to test files
+regardless of location.
+
+::
+
+    % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv --param swift_site_config=${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/lit.site.cfg ${SWIFT_SOURCE_ROOT}/test/Parse/
+
+For more complicated configuration, copy the invocation from one of the build
+targets mentioned above and modify it as necessary. lit.py also has several
+useful features, like timing tests and providing a timeout. Check these features
+out with ``lit.py -h``.
+
+Writing tests
+-------------
+
+General guidelines
+^^^^^^^^^^^^^^^^^^
+
+When adding a new testcase, try to find an existing test file focused on the
+same topic rather than starting a new test file.  There is a fixed runtime cost
+for every test file.  On the other hand, avoid dumping new tests in a file that
+is only remotely related to the purpose of the new tests.
+
+Don't limit a test to a certain platform or hardware configuration just because
+this makes the test slightly easier to write.  This sometimes means a little
+bit more work when adding the test, but the payoff from the increased testing
+is significant.  We heavily rely on portable tests to port Swift to other
+platforms.
+
+Avoid using unstable language features in tests which test something else (for
+example, avoid using an unstable underscored attribute when another
+non-underscored attribute would work).
+
+Avoid using arbitrary implementation details of the standard library.  Always
+prefer to define types locally in the test, if feasible.
+
+Avoid purposefully shadowing names from the standard library, this makes the
+test extremely confusing (if nothing else, to understand the intent --- was the
+compiler bug triggered by this shadowing?)  When reducing a compiler testcase
+from the standard library source, rename the types and APIs in the testcase to
+differ from the standard library APIs.
+
+In IRGen, SILGen and SIL tests, avoid using platform-dependent implementation
+details of the standard library (unless doing so is point of the test).
+Platform-dependent details include:
+
+* ``Int`` (use integer types with explicit types instead).
+
+* Layout of ``String``, ``Array``, ``Dictionary``, ``Set``.  These differ
+  between platforms that have Objective-C interop and those that don't.
+
+Unless testing the standard library, avoid using arbitrary standard library
+types and APIs, even if it is very convenient for you to do so in your tests.
+Using the more common APIs like ``Array`` subscript or ``+`` on ``IntXX`` is
+acceptable.  This is important because you can't rely on the full standard
+library being available.  The long-term plan is to introduce a mock, minimal
+standard library that only has a very basic set of APIs.
+
+If you write an executable test please add ``REQUIRES: executable_test`` to the
+test.
+
+Substitutions in lit tests
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Substitutions that start with ``%target`` configure the compiler for building
+code for the target that is not the build machine:
+
+* ``%target-parse-verify-swift``: parse and type check the current Swift file
+  for the target platform and verify diagnostics, like ``swift -parse -verify
+  %s``.
+
+  Use this substitution for testing semantic analysis in the compiler.
+
+* ``%target-swift-frontend``: run ``swift -frontend`` for the target.
+
+  Use this substitution (with extra arguments) for tests that don't fit any
+  other pattern.
+
+* ``%target-swift-frontend(mock-sdk:`` *mock sdk arguments* ``)`` *other
+  arguments*: like ``%target-swift-frontend``, but allows to specify command
+  line parameters (typically ``-sdk`` and ``-I``) to use a mock SDK and SDK
+  overlay that would take precedence over the target SDK.
+
+* ``%target-build-swift``: compile and link a Swift program for the target.
+
+  Use this substitution only when you intend to run the program later in the
+  test.
+
+* ``%target-run-simple-swift``: build a one-file Swift program and run it on
+  the target machine.
+
+  Use this substitution for executable tests that don't require special
+  compiler arguments.
+
+  Add ``REQUIRES: executable_test`` to the test.
+
+* ``%target-run-stdlib-swift``: like ``%target-run-simple-swift`` with
+  ``-parse-stdlib -Xfrontend -disable-access-control``.
+
+  This is sometimes useful for testing the Swift standard library.
+
+  Add ``REQUIRES: executable_test`` to the test.
+
+* ``%target-repl-run-simple-swift``: run a Swift program in a REPL on the
+  target machine.
+
+* ``%target-run``: run a command on the target machine.
+
+  Add ``REQUIRES: executable_test`` to the test.
+
+* ``%target-jit-run``: run a Swift program on the target machine using a JIT
+  compiler.
+
+* ``%target-swiftc_driver``: FIXME
+
+* ``%target-sil-opt``: run ``sil-opt`` for the target.
+
+* ``%target-sil-extract``: run ``sil-extract`` for the target.
+
+* ``%target-swift-ide-test``: run ``swift-ide-test`` for the target.
+
+* ``%target-swift-ide-test(mock-sdk:`` *mock sdk arguments* ``)`` *other
+  arguments*: like ``%target-swift-ide-test``, but allows to specify command
+  line parameters to use a mock SDK.
+
+* ``%target-swiftc_driver``: FIXME.
+
+* ``%target-swift-autolink-extract``: run ``swift-autolink-extract`` for the
+  target to extract its autolink flags on platforms that support them (when the
+  autolink-extract feature flag is set)
+
+* ``%target-clang``: run the system's ``clang++`` for the target.
+
+  If you want to run the ``clang`` executable that was built alongside 
+  Swift, use ``%clang`` instead.
+
+* ``%target-ld``: run ``ld`` configured with flags pointing to the standard
+  library directory for the target.
+
+* ``%target-cc-options``: the clang flags to setup the target with the right
+  architecture and platform version.
+
+Always use ``%target-*`` substitutions unless you have a good reason.  For
+example, an exception would be a test that checks how the compiler handles
+mixing module files for incompatible platforms (that test would need to compile
+Swift code for two different platforms that are known to be incompatible).
+
+When you can't use ``%target-*`` substitutions, you can use:
+
+* ``%swift_driver_plain``: FIXME.
+* ``%swiftc_driver_plain``: FIXME.
+* ``%swift_driver``: FIXME.
+* ``%swiftc_driver``: FIXME.
+* ``%sil-opt``: FIXME.
+* ``%sil-extract``: FIXME.
+* ``%lldb-moduleimport-test``: FIXME.
+* ``%swift-ide-test_plain``: FIXME.
+* ``%swift-ide-test``: FIXME.
+* ``%llvm-opt``: FIXME.
+* ``%swift``: FIXME.
+* ``%clang-include-dir``: FIXME.
+* ``%clang-importer-sdk``: FIXME.
+
+Other substitutions:
+
+* ``%leaks-runner``: FIXME.
+* ``%clang_apinotes``: FIXME.
+* ``%clang``: FIXME.
+* ``%target-triple``: FIXME, possible values.
+* ``%target-cpu``: FIXME, possible values.
+* ``%target-os``: FIXME, possible values.
+* ``%target-object-format``: the platform's object format (elf, macho, coff).
+* ``%target-runtime``: the platform's Swift runtime (objc, native).
+* ``%target-ptrsize``: the pointer size of the target (32, 64).
+* ``%sdk``: FIXME.
+* ``%gyb``: FIXME.
+
+* ``%platform-module-dir``: absolute path of the directory where the stdandard
+  library module file for the target platform is stored.  For example,
+  ``/.../lib/swift/macosx``.
+
+* ``%platform-sdk-overlay-dir``: absolute path of the directory where the SDK
+  overlay module files for the target platform are stored.
+
+* ``%target-swiftmodule-name`` and ``%target-swiftdoc-name``: the basename of
+  swiftmodule and swiftdoc files for a framework compiled for the target (for
+  example, ``arm64.swiftmodule`` and ``arm64.swiftdoc``).
+
+* ``%target-sdk-name``: only for Apple platforms: ``xcrun``-style SDK name
+  (``macosx``, ``iphoneos``, ``iphonesimulator``).
+
+When writing a test where output (or IR, SIL) depends on the bitness of the
+target CPU, use this pattern::
+
+  // RUN: %target-swift-frontend ... | FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-ptrsize %s
+
+  // CHECK: common line
+  // CHECK-32: only for 32-bit
+  // CHECK-64: only for 64-bit
+
+  // FileCheck does a single pass for a combined set of CHECK lines, so you can
+  // do this:
+  //
+  // CHECK: define @foo() {
+  // CHECK-32: integer_literal $Builtin.Int32, 0
+  // CHECK-64: integer_literal $Builtin.Int64, 0
+
+When writing a test where output (or IR, SIL) depends on the target CPU itself,
+use this pattern::
+
+  // RUN: %target-swift-frontend ... | FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-cpu %s
+
+  // CHECK: common line
+  // CHECK-i386:   only for i386
+  // CHECK-x86_64: only for x86_64
+  // CHECK-armv7:  only for armv7
+  // CHECK-arm64:  only for arm64
+
+Features for ``REQUIRES`` and ``XFAIL``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+FIXME: list.
+
+When writing a test specific to x86, if possible, prefer ``REQUIRES:
+CPU=i386_or_x86_64`` to ``REQUIRES: CPU=x86_64``.
+
+``swift_test_mode_optimize[_unchecked|none]`` and
+``swift_test_mode_optimize[_unchecked|none]_<CPUNAME>`` to specify a test mode
+plus cpu configuration.
+
+``optimized_stdlib_<CPUNAME>``` to specify a optimized stdlib plus cpu
+configuration.
+
+Feature ``REQUIRES: executable_test``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This feature marks an executable test. The test harness makes this feature
+generally available. It can be used to restrict the set of tests to run.
+
+StdlibUnittest
+^^^^^^^^^^^^^^
+
+Tests accept command line parameters, run StdlibUnittest-based test binary
+with ``--help`` for more information.
+
+Testing memory management in execution tests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In execution tests, memory management testing should be performed
+using local variables enclosed in a closure passed to the standard
+library ``autoreleasepool`` function. For example::
+
+  // A counter that's decremented by Canary's deinitializer.
+  var CanaryCount = 0
+
+  // A class whose instances increase a counter when they're destroyed.
+  class Canary {
+    deinit { ++CanaryCount }
+  }
+
+  // Test that a local variable is correctly released before it goes out of
+  // scope.
+  CanaryCount = 0
+  autoreleasepool {
+    let canary = Canary()
+  }
+  assert(CanaryCount == 1, "canary was not released")
+
+Memory management tests should be performed in a local scope because Swift does
+not guarantee the destruction of global variables. Code that needs to
+interoperate with Objective-C may put references in the autorelease pool, so
+code that uses an ``if true {}`` or similar no-op scope instead of
+``autoreleasepool`` may falsely report leaks or fail to catch overrelease bugs.
+If you're specifically testing the autoreleasing behavior of code, or do not
+expect code to interact with the Objective-C runtime, it may be OK to use ``if
+true {}``, but those assumptions should be commented in the test.