[docs] Add an external description of the swiftc driver. (#4992)

Aimed at people writing build systems who won't be satisfied with
"just use xcodebuild" or "just use SwiftPM", mostly because they'll
go ahead doing what they're doing anyway and it'll be incorrect.

Author: jrose-apple

docs/Driver.md

@@ -0,0 +1,319 @@
+# The Swift Driver, Compilation Model, and Command-Line Experience
+
+_or, "why can't I only compile the files that changed?"_
+
+The Swift compiler's command-line interface resembles that of other compilers,
+particularly GCC and Clang. However, Swift's compilation model and some of its
+language features make it a bit tricky to plug into a larger build system. In
+particular, there's no correct way to specify a "one command per file" build
+rule for a normal Swift module.
+
+("What?" I know! Read on.)
+
+The target audience for this document is people who want to integrate the Swift
+compiler into their build system, rather than using Xcode or the package
+manager (`swift build`). If you're looking to work on the driver itself...well,
+this is probably still useful to you, but you should also check out
+DriverInternals.rst and maybe DependencyAnalysis.rst as well. If you're just
+using Xcode or SwiftPM and want to find out what mysterious command-line
+options you could be passing, `swiftc --help` is a better choice.
+
+If you're invoking `swift -frontend` directly, and you aren't working on the
+compiler itself...well, this document should convince you to not do that.
+
+Some terms:
+
+- For the purposes of this document, a _module_ is a single distributable unit
+  of API. (We use this term for a lot of other things too, though; check out
+  Lexicon.rst for the full list.) "Foundation" is a single module, as is the
+  Swift standard library ("Swift"). An app is a module too.
+
+- A _compilation unit_ is a set of source files that are compiled together. In
+  Swift, everything intended to be in the same module must be part of the same
+  compilation unit, and every file in the same compilation unit are assumed to
+  be part of the same module. Doing anything else is unsupported.
+
+- The _driver_ is the program that's run when you invoke `swift` or `swiftc`.
+  This doesn't actually compile anything itself; instead it invokes other tools
+  to produce the desired output.
+
+- The _frontend_ is the program that actually compiles code (and in interpreter
+  mode, executes it via a JIT). It's hidden behind `swift -frontend`.
+
+  The frontend is an implementation detail. No aspects of its command-line
+  interface are considered stable. Even its existence isn't guaranteed.
+
+- The _REPL_ is a mode of the debugger (LLDB) that is launched by the driver
+  when you invoke `swift` with no inputs. It doesn't actually come up again in
+  this document, but it's probably worth mentioning for completeness.
+
+
+## What gets run? ##
+
+Part of Swift's model is that a file can implicitly see entities declared
+elsewhere in the same module as long as they have not been marked private.
+Consequently, compiling any one file needs some knowledge of all the others.
+However, we don't want a build model that rules out incremental builds, nor one
+that forces non-parallel compilation. Consequently, the Swift driver has three
+stages of subprocesses:
+
+1. "Frontend jobs"
+2. "Module merging"
+3. Linking
+
+Dependencies between the subprocesses are managed by the driver. Outputs are
+controlled by `-o` and other various compiler flags; see `swiftc --help` for
+more information.
+
+
+### Frontend jobs ###
+
+A normal Swift compilation starts off with as many _frontend jobs_ as input
+files. Each invocation of the Swift frontend parses every file in the module,
+but also has a particular file marked as the _primary file._ A job is only
+responsible for compiling its primary file, and only does as much work as it
+needs to to compile that file, lazily type-checking declarations in other files
+in the module.
+
+A frontend job emits diagnostics, an object file, dependency information (see
+"Incremental Builds" below), and a _partial module file._
+
+
+### Module merging ###
+
+Swift doesn't have header files; instead it uses a generated binary format
+called a _Swift module file_ (.swiftmodule). With N frontend jobs, it becomes
+necessary to stitch all the partial module files together. This job is also
+performed by the frontend in a step called "module merging". Module merging
+only produces a single merged module file (and accompanying documentation file
+with the extension .swiftdoc), which can then be imported in later builds.
+
+The Swift module file format is not stable; using it across compiler versions
+is not supported. It also currently includes a ton of private information about
+both your module and your compilation environment, so make sure you don't
+distribute them with your final build products.
+
+
+### Linking ###
+
+The last stage of compilation is linking. In addition to actually invoking the
+linker, the compiler needs to accomplish two other tasks:
+
+- **Autolinking:** Swift object files encode information about what libraries
+  they depend on. On Apple platforms the linker can read this information
+  directly; on other platforms it's extracted using the
+  `swift-autolink-extract` helper tool. Of course the build system can also
+  provide manual link commands too.
+
+- **Debugging:** In addition to the usual DWARF format, interactive debugging
+  of a Swift program requires access to its module. This is supported via a
+  custom linker flag on Apple platforms and by the `swift-modulewrap` tool
+  elsewhere. Apple platforms also invoke `dsymutil` after linking to produce a
+  dSYM debugging archive, so that the program is still debuggable even once the
+  object files have been cleaned.
+
+
+## Output File Maps ##
+
+> There are three numbers in computer science: 0, 1, and N.
+
+The problem with running a separate frontend job for each input file is that it
+suddenly becomes much more complicated to determine where outputs go. If you're
+just going to link everything together at the end this isn't really a problem,
+but a lot of build systems just want the compiler to produce object files. If
+you want to use Swift's cross-file dependency tracking, or get Clang-style
+serialized diagnostics (.dia files) instead of just text output, those are also
+generated per-input-file. And some build systems (\*cough\*Xcode\*cough\*) want
+to individually track persisted output files, instead of dumping them into a
+temporary directory. Trying to specify the outputs for every input file on the
+command line would be awkward and ridiculous, so instead we use an _output file
+map._
+
+An output file map contains a JSON dictionary that looks like this:
+
+```json
+{
+  "/path/to/src/foo.swift": {
+    "object": "/path/to/build/foo.o",
+    "dependencies": "/path/to/build/foo.d",
+    "swift-dependencies": "/path/to/build/foo.swiftdeps",
+    "diagnostics": "/path/to/build/foo.dia"
+  },
+  "/path/to/src/bar.swift": {
+    "object": "/path/to/build/bar.o",
+    "dependencies": "/path/to/build/bar.d",
+    "swift-dependencies": "/path/to/build/bar.swiftdeps",
+    "diagnostics": "/path/to/build/bar.dia"
+  },
+  "": {
+    "swift-dependencies": "/path/to/build/main-build-record.swiftdeps"
+  }
+}
+```
+
+The build system is responsible for generating this file. The input file names
+don't have to be absolute paths, but they do have to match the form used in the
+invocation of the compiler. No canonicalization is performed. The special `""`
+entry is used for outputs that apply to the entire build.
+
+The "dependencies" entries refer to Make-style dependency files (similar to GCC
+and Clang's `-MD` option), which can be used to track cross-module and header
+file dependencies. "swift-dependencies" entries are required to perform
+incremental builds (see below). The "diagnostics" entries are only for your own
+use; if you don't need Clang-style serialized diagnostics they can be omitted.
+
+The output file map accepts other entries, but they should not be considered
+stable. Please stick to what's shown here.
+
+(Note: In the example output file map above, all of the per-file outputs are
+being emitted to the same directory. [SR-327][] covers adding a flag that would
+infer this behavior given a directory path.)
+
+  [SR-327]: https://bugs.swift.org/browse/SR-327
+
+
+## Whole-Module Optimization ##
+
+When the `-whole-module-optimization` flag is passed to Swift, the compilation
+model changes significantly. In this mode, the driver only invokes the frontend
+once, and there is no primary file. Instead, every file is parsed and
+type-checked, and then all generated code is optimized together.
+
+Whole-module builds actually have two modes: threaded and non-threaded. The
+default is non-threaded, which produces a single object file for the entire
+compilation unit (.o). If you're just producing object files, you can use the
+usual `-o` option with `-c` to control where the single output goes.
+
+In threaded mode, one object file is produced for every input source file, and
+the "backend" processing of the generated code (basically, everything that's
+not Swift-language-specific) is performed on multiple threads. Like
+non-whole-module compilation, the locations of the object files is controlled
+by the output file map. Only one file is produced for each non-object output,
+however. The location of these additional outputs is controlled by command-line
+options, except for Make-style dependencies and serialized diagnostics, which
+use an entry under `""` in the output file map.
+
+Threaded mode is controlled by the `-num-threads` command-line option rather
+than `-j` used to control the number of jobs (simultaneous subprocesses spawned
+by the driver). Why? While changing the number of jobs should never affect the
+final product, using threaded vs. non-threaded compilation does.
+
+Specifics of whole-module optimization mode are subject to change, particularly
+in becoming more like non-whole-module builds.
+
+
+## Incremental Builds ##
+
+Incremental builds in Swift work by primarily by cross-file dependency
+analysis, described in DependencyAnalysis.rst. Compiling a single file might be
+necessary because that file has changed, but it could also be because that file
+depends on something else that might have changed. From a build system
+perspective, the files in a particular module can't be extracted from each
+other; a top-level invocation of the compiler will result in a valid
+compilation of the entire module, but manually recompiling certain files is not
+guaranteed to do anything sensible.
+
+Performing an incremental build is easy; just pass `-incremental` and be sure to
+put "swift-dependencies" entries in your output file map.
+
+Incremental builds don't mix with whole-module builds, which get their
+whole-module-ness by rebuilding everything every time *just in case* there's
+some extra optimizations that can be made by repeated inlining, or by *really*
+being sure that a method is never overridden.
+
+(Can this sort of information be cached? Of course. But there's a question of
+whether it's going to be easier to make whole-module builds be more incremental
+or to make incremental builds support more cross-file optimization.)
+
+A particular nasty bit about incremental builds: because in general the
+compiler only knows that a file *has* changed and not *what* changed, it's
+possible that the driver will start off assuming that some file won't need to
+be recompiled, and then discover later on that it should be. This means that
+it's not possible to use classic build models like Makefiles to make builds
+incremental, because Makefiles don't accomodate dependency graph changes during
+the build.
+
+Only the "frontend" tasks are considered skippable; module-merging and linking
+always occurs. (It is assumed that if *none* of the inputs changed, the
+build system wouldn't have invoked the compiler at all.)
+
+Currently Swift does not provide any way to report tasks to be compiled out to
+some other build system. This is definitely something we're interested in,
+though---it would enable better integration between the compiler and llbuild
+(the build system used by the package manager), which would lead to faster
+compile times when many dependencies were involved.
+
+The format of the "swiftdeps" files used to track cross-file dependencies
+should be considered fragile and opaque; it is not guaranteed to remain stable
+across compiler versions.
+
+
+## `-embed-bitcode` ##
+
+Apple's "Embed Bitcode" feature adds an extra bit of complexity: to be sure
+builds from the embedded bitcode are identical to those built locally, the
+driver actually splits compilation in two. The first half of compilation
+produces LLVM IR, then the driver runs so-called _backend jobs,_ which compile
+the IR the rest of the way into binaries.
+
+There's not really any secrets here, since it's all there in the source, but
+the particular mechanism isn't guaranteed by any means---Apple could decide to
+change it whenever they want. It's probably best not to special-case any logic
+here; just using the driver as intended should work fine.
+
+
+## So, how should I work Swift into my build system? ##
+
+Step 0 is to see if you can use the Swift package manager instead. If so, it's
+mostly just `swift build`. But if you're reading this document you're probably
+past that, so:
+
+1. Generate an output file map that contains all the per-file outputs you care
+   about. Most likely this is just the object files and incremental build
+   dependency files; everything else is an intermediate. (There should probably 
+   be a tool that does this, perhaps built on what the package manager does.)
+
+2. Set TMPDIR to somewhere you don't mind uninteresting intermediate files 
+   going.
+
+3. Do one of the following:
+
+   - Invoke `swiftc -emit-executable` or `swiftc -emit-library`. Pass all the
+     Swift files, even if you're building incrementally (`-incremental`). Pass
+     `-j <N>` and/or `-num-threads <N>` if you have cores to spare. Pass
+     `-emit-module-path <path>` if you're building a library that you need to
+     import later.
+
+     If you want debugging that's more than `-gline-tables-only`, this is the
+     only supported way to do it today. ([SR-2637][] and [SR-2660][] are aimed
+     at improving on this.) On the plus side, this mode doesn't strictly need
+     an output file map if you give up incremental builds.
+
+   - Invoke `swiftc -c`, then pass the resulting object files to your linker.
+     All the same options from above apply, but you'll have to manually deal
+     with the work the compiler would have done automatically for you.
+
+   - Invoke `swiftc -c` with `-###`. Then run all of the outputted commands
+     that include `-primary-file`, then run the remaining commands in order
+     (they may have dependencies). If none of the commands have `-primary-file`
+     in them, they're not parallelizable, sorry.
+
+     This is the most hacky approach, because (a) it involves looking for an
+     internal flag in a non-stable interface, and (b) you don't get anything
+     incremental out of this. We could stand some improvements here.
+
+   Whatever you do, do *not* invoke the frontend directly.
+
+
+### Questions ###
+
+_Can I link all the object files together in the same binary, even if they came
+from multiple modules?_
+
+This is not currently supported, and debugging probably won't work. (See
+[SR-2637][] and [SR-2660][] for more details.) However, if you are using
+`-gnone` or `-gline-tables-only`, the worst you will suffer is more symbols
+being visible than are strictly necessary.
+
+  [SR-2637]: https://bugs.swift.org/browse/SR-2637
+  [SR-2660]: https://bugs.swift.org/browse/SR-2660