Document Some Whys

* Document a number of legacy conditions and edge cases

* Add lexicon definitions for "dependency source", "dependency sink",
"cascading dependency" and "private dependency"

Author: CodaFi

docs/Lexicon.rst

@@ -77,6 +77,17 @@ source code, tests, and commit messages. See also the `LLVM lexicon`_.
     same; the exception is when generics get involved. In this case you'll need
     a `generic environment`. Contrast with `sugared type`.
 
+  cascading dependency
+    A kind of dependency edge relevant to the incremental name tracking
+    subsystem. A cascading dependency (as opposed to a
+    `private dependency <private dependency>`) requires the Swift driver to
+    transitively consider dependency edges in the file that defines the used
+    name when incremental compilation is enabled. A cascading dependency is much
+    safer to produce than its private counterpart, but it comes at the cost of
+    increased usage of compilation resources - even if those resources are being
+    wasted on rebuilding a file that didn't actually require rebuilding.
+    See :doc:`DependencyAnalysis.rst <DependencyAnalysis>`.
+
   Clang importer
     The part of the compiler that reads C and Objective-C declarations and
     exposes them as Swift. Essentially contains a small instance of Clang
@@ -100,6 +111,24 @@ source code, tests, and commit messages. See also the `LLVM lexicon`_.
     i.e. one that conforming types don't *have* to implement but have the option
     to "customize".
 
+  dependency sink
+    Any request that uses a matching dependency source to write dependency
+    edges into the referenced name trackers. For example, a request that
+    performs direct lookup will write the name being looked up into the
+    name tracker associated with the file that issued the lookup request.
+    The request evaluator automatically determines the appropriate tracker
+    for a dependency sink to write into based on the current active
+    `dependency source <dependency source>` request.
+
+  dependency source
+    Any request that defines a scope under which reference dependencies may be
+    registered. For example, a request to type check an entire file is a
+    dependency source. Dependency sources are automatically managed by the
+    request evaluator as request evaluation proceeds. Dependency sources provide
+    one half of the necessary information to complete a full dependency edge.
+    The other half is provided by corresponding
+    `dependency sink <dependency sink>` requests.
+
   DI (definite initialization / definitive initialization)
     The feature that no uninitialized variables, constants, or properties will
     be read by a program, or the analysis pass that operates on SIL to
@@ -327,6 +356,17 @@ source code, tests, and commit messages. See also the `LLVM lexicon`_.
     only needed for context. See also
     `Whole-Module Optimization <WMO (whole-module optimization)>`.
 
+  private dependency
+    A kind of dependency edge relevant to the incremental name tracking
+    subsystem. A private dependency (as opposed to a
+    `cascading dependency <cascading dependency>`) declares a dependency edge
+    from one file to a name referenced in that file that does not
+    require further transitive evaluation of dependency edges by the Swift
+    driver. Private dependencies are therefore cheaper than cascading
+    dependencies, but must be used with the utmost care or dependent files will
+    fail to rebuild and the result will most certainly be a miscompile.
+    See :doc:`DependencyAnalysis.rst <DependencyAnalysis>`.
+
   QoI
     "Quality of implementation." The term is meant to describe not how
     well-engineered a particular implementation is, but how much value it

include/swift/AST/Evaluator.h

@@ -483,7 +483,7 @@ class Evaluator {
   /// If there is an active dependency source, returns its
   /// \c ReferencedNameTracker. Else, returns \c nullptr.
   ReferencedNameTracker *getActiveDependencyTracker() const {
-    if (auto *source = getActiveDependencySource())
+    if (auto *source = getActiveDependencySourceOrNull())
       return source->getRequestBasedReferencedNameTracker();
     return nullptr;
   }
@@ -508,7 +508,12 @@ class Evaluator {
 
   /// Returns the active dependency's source file, or \c nullptr if no
   /// dependency source is active.
-  SourceFile *getActiveDependencySource() const {
+  ///
+  /// The use of this accessor is strongly discouraged, as it implies that a
+  /// dependency sink is seeking to filter out names based on the files they
+  /// come from. Existing callers are being migrated to more reasonable ways
+  /// of judging the relevancy of a dependency.
+  SourceFile *getActiveDependencySourceOrNull() const {
     if (dependencySources.empty())
       return nullptr;
     return dependencySources.back().getPointer();

include/swift/AST/EvaluatorDependencies.h

@@ -61,6 +61,24 @@ enum class DependencyScope : bool {
 };
 
 /// Returns a \c DependencyScope appropriate for the given (formal) access level.
+///
+/// :warning: This function exists to bridge the old manual reference
+/// dependencies code to the new evaluator-based reference dependencies code.
+/// The manual code often made private/cascading scope judgements based on the
+/// access level of a declaration. While this makes some sense intuitively, it
+/// does not necessarily capture an accurate picture of where real incremental
+/// dependencies lie. For example, references to formally private types can
+/// "escape" to contexts that have no reference to the private name if, say,
+/// the layout of that private type is taken into consideration by
+/// SILGen or IRGen in a separate file that references the declaration
+/// transitively. However, due to the density of the current dependency
+/// graph, redundancy in registered dependency edges, and the liberal use of
+/// cascading edges, we may be saved from the worst consequences of this
+/// modelling choice.
+///
+/// The use of access-levels for dependency decisions is an anti-pattern that
+/// should be revisited once finer-grained dependencies are explored more
+/// thoroughly.
 inline DependencyScope getScopeForAccessLevel(AccessLevel l) {
   switch (l) {
   case AccessLevel::Private:

include/swift/AST/SimpleRequest.h

@@ -46,13 +46,19 @@ enum class RequestFlags {
   ///
   /// This bit is optional.  High-level requests
   /// (e.g. \c TypeCheckSourceFileRequest) will require it.
+  ///
+  /// For further discussion on incremental dependencies
+  /// see DependencyAnalysis.rst.
   DependencySource = 1 << 3,
   /// This request introduces the sink component of a source-sink
   /// incremental dependency pair and is a consumer of the current
   /// dependency scope.
   ///
   /// This bit is optional. Name lookup requests
   /// (e.g. \c DirectLookupRequest) will require it.
+  ///
+  /// For further discussion on incremental dependencies
+  /// see DependencyAnalysis.rst.
   DependencySink = 1 << 4,
 };
 

lib/AST/NameLookupRequests.cpp

@@ -98,7 +98,7 @@ InheritedProtocolsRequest::readDependencySource(Evaluator &e) const {
   if (!PD->getParentSourceFile())
     return { nullptr, e.getActiveSourceScope() };
   return {
-    e.getActiveDependencySource(),
+    e.getActiveDependencySourceOrNull(),
     evaluator::getScopeForAccessLevel(PD->getFormalAccess())
   };
 }
@@ -186,7 +186,7 @@ void ExtendedNominalRequest::writeDependencySink(
   auto *SF = std::get<0>(getStorage())->getParentSourceFile();
   if (!SF)
     return;
-  if (SF != eval.getActiveDependencySource())
+  if (SF != eval.getActiveDependencySourceOrNull())
     return;
   tracker.addUsedMember({value, Identifier()},
                         eval.isActiveSourceCascading());
@@ -217,7 +217,7 @@ GetDestructorRequest::readDependencySource(Evaluator &eval) const {
   // valid 'deinit' declaration cannot occur outside of the
   // definition of a type.
   return {
-    eval.getActiveDependencySource(),
+    eval.getActiveDependencySourceOrNull(),
     evaluator::DependencyScope::Private
   };
 }
@@ -393,7 +393,7 @@ void LookupConformanceInModuleRequest::writeDependencySink(
   if (!Adoptee)
     return;
 
-  auto *source = eval.getActiveDependencySource();
+  auto *source = eval.getActiveDependencySourceOrNull();
   assert(source && "Missing dependency source?");
 
   // Decline to record conformances defined outside of the active module.

lib/AST/TypeCheckRequests.cpp

@@ -161,7 +161,7 @@ evaluator::DependencySource
 SuperclassTypeRequest::readDependencySource(Evaluator &e) const {
   const auto access = std::get<0>(getStorage())->getFormalAccess();
   return {
-    e.getActiveDependencySource(),
+    e.getActiveDependencySourceOrNull(),
     evaluator::getScopeForAccessLevel(access)
   };
 }
@@ -1348,7 +1348,7 @@ LookupAllConformancesInContextRequest::readDependencySource(
     const NominalTypeDecl *nominal = ext->getExtendedNominal();
     if (!nominal) {
       return {
-        eval.getActiveDependencySource(),
+        eval.getActiveDependencySourceOrNull(),
         evaluator::DependencyScope::Cascading
       };
     }
@@ -1357,7 +1357,7 @@ LookupAllConformancesInContextRequest::readDependencySource(
     defaultAccess = cast<NominalTypeDecl>(dc)->getFormalAccess();
   }
   return {
-    eval.getActiveDependencySource(),
+    eval.getActiveDependencySourceOrNull(),
     evaluator::getScopeForAccessLevel(defaultAccess)
   };
 }