[Evaluator] Online Request-Based Incremental Dependency Tracking

docs/Lexicon.rst

@@ -77,6 +77,17 @@ source code, tests, and commit messages. See also the `LLVM lexicon`_.
     same; the exception is when generics get involved. In this case you'll need
     a `generic environment`. Contrast with `sugared type`.
 
+  cascading dependency
+    A kind of dependency edge relevant to the incremental name tracking
+    subsystem. A cascading dependency (as opposed to a
+    `private dependency <private dependency>`) requires the Swift driver to
+    transitively consider dependency edges in the file that defines the used
+    name when incremental compilation is enabled. A cascading dependency is much
+    safer to produce than its private counterpart, but it comes at the cost of
+    increased usage of compilation resources - even if those resources are being
+    wasted on rebuilding a file that didn't actually require rebuilding.
+    See :doc:`DependencyAnalysis.rst <DependencyAnalysis>`.
+
   Clang importer
     The part of the compiler that reads C and Objective-C declarations and
     exposes them as Swift. Essentially contains a small instance of Clang
@@ -100,6 +111,24 @@ source code, tests, and commit messages. See also the `LLVM lexicon`_.
     i.e. one that conforming types don't *have* to implement but have the option
     to "customize".
 
+  dependency sink
+    Any request that uses a matching dependency source to write dependency
+    edges into the referenced name trackers. For example, a request that
+    performs direct lookup will write the name being looked up into the
+    name tracker associated with the file that issued the lookup request.
+    The request evaluator automatically determines the appropriate tracker
+    for a dependency sink to write into based on the current active
+    `dependency source <dependency source>` request.
+
+  dependency source
+    Any request that defines a scope under which reference dependencies may be
+    registered. For example, a request to type check an entire file is a
+    dependency source. Dependency sources are automatically managed by the
+    request evaluator as request evaluation proceeds. Dependency sources provide
+    one half of the necessary information to complete a full dependency edge.
+    The other half is provided by corresponding
+    `dependency sink <dependency sink>` requests.
+
   DI (definite initialization / definitive initialization)
     The feature that no uninitialized variables, constants, or properties will
     be read by a program, or the analysis pass that operates on SIL to
@@ -327,6 +356,17 @@ source code, tests, and commit messages. See also the `LLVM lexicon`_.
     only needed for context. See also
     `Whole-Module Optimization <WMO (whole-module optimization)>`.
 
+  private dependency
+    A kind of dependency edge relevant to the incremental name tracking
+    subsystem. A private dependency (as opposed to a
+    `cascading dependency <cascading dependency>`) declares a dependency edge
+    from one file to a name referenced in that file that does not
+    require further transitive evaluation of dependency edges by the Swift
+    driver. Private dependencies are therefore cheaper than cascading
+    dependencies, but must be used with the utmost care or dependent files will
+    fail to rebuild and the result will most certainly be a miscompile.
+    See :doc:`DependencyAnalysis.rst <DependencyAnalysis>`.
+
   QoI
     "Quality of implementation." The term is meant to describe not how
     well-engineered a particular implementation is, but how much value it

docs/RequestEvaluator.md

@@ -52,6 +52,11 @@ The request-evaluator contains a cache of any requests that have already been ev
 
 Until then, the request-evaluator lives in a compiler that has mutable ASTs, and will for the foreseeable future. To cope with this, requests can opt for "separate" caching of their results, implementing a simple protocol to query their own cache (`getCachedResult`) and set the value for the cache (`cacheResult`). For now, these functions can directly modify state in the AST, allowing the requests to be mixed with direct mutation of the state. For each request, the intent is to make all state access go through the evaluator, but getting there can be an incremental process.
 
+## Incremental Dependency Tracking
+Request evaluation naturally captures the dependency structure of any given invocation of the compiler frontend. In fact, it captures it so well that the request graph trace generated by a select kind of lookup request can be used to completely recover the information relevant to the Swift compiler's incremental compilation subsystem. For these select *dependency-relevant* requests, we can further subdivide them into so-called *dependency sources* and *dependency sinks*. A dependency source is any (usually high-level) request that introduces a new context under which dependencies can be registered. Currently, these are the requests that operate that the level of individual source files. A dependency sink is any (usually lower-level) request that executes as a sub-computation of a dependency source. Any names that are dependency-relevant, such as the result of a lookup in a particular context, are then registered against trackers in the active dependency source (file). Using this, the evaluator pushes and pops sources and sinks automatically as request evaluation proceeds, and sink requests pair automatically to source requests to write out dependency information.
+
+To define a request as a dependency source, it must implement an accessor for the new active scope (`readDependencySource`). To define a request as a dependency sink, it must implement a function that writes the result of evaluating the request into the current active source (`writeDependencySink`).
+
 ## Open Projects
 
 The request-evaluator is relatively new to the Swift compiler, having been introduced in mid-2018. There are a number of improvements that can be made to the evaluator itself and how it is used in the compiler:

include/swift/AST/AccessRequests.h

@@ -33,7 +33,7 @@ class ValueDecl;
 class AccessLevelRequest :
     public SimpleRequest<AccessLevelRequest,
                          AccessLevel(ValueDecl *),
-                         CacheKind::SeparatelyCached> {
+                         RequestFlags::SeparatelyCached> {
 public:
   using SimpleRequest::SimpleRequest;
 
@@ -56,7 +56,7 @@ class AccessLevelRequest :
 class SetterAccessLevelRequest :
     public SimpleRequest<SetterAccessLevelRequest,
                          AccessLevel(AbstractStorageDecl *),
-                         CacheKind::SeparatelyCached> {
+                         RequestFlags::SeparatelyCached> {
 public:
   using SimpleRequest::SimpleRequest;
 
@@ -79,7 +79,7 @@ using DefaultAndMax = std::pair<AccessLevel, AccessLevel>;
 class DefaultAndMaxAccessLevelRequest :
     public SimpleRequest<DefaultAndMaxAccessLevelRequest,
                          DefaultAndMax(ExtensionDecl *),
-                         CacheKind::SeparatelyCached> {
+                         RequestFlags::SeparatelyCached> {
 public:
   using SimpleRequest::SimpleRequest;
 private:

include/swift/AST/Evaluator.h

@@ -19,6 +19,7 @@
 #define SWIFT_AST_EVALUATOR_H
 
 #include "swift/AST/AnyRequest.h"
+#include "swift/AST/EvaluatorDependencies.h"
 #include "swift/Basic/AnyValue.h"
 #include "swift/Basic/Debug.h"
 #include "swift/Basic/Defer.h"
@@ -47,6 +48,12 @@ class DiagnosticEngine;
 class Evaluator;
 class UnifiedStatsReporter;
 
+namespace detail {
+// Remove this when the compiler bumps to C++17.
+template <typename...>
+using void_t = void;
+}
+
 /// An "abstract" request function pointer, which is the storage type
 /// used for each of the
 using AbstractRequestFunction = void(void);
@@ -224,6 +231,38 @@ class Evaluator {
   /// so all clients must cope with cycles.
   llvm::DenseMap<AnyRequest, std::vector<AnyRequest>> dependencies;
 
+  /// A stack of dependency sources in the order they were evaluated.
+  llvm::SmallVector<evaluator::DependencySource, 8> dependencySources;
+
+  /// An RAII type that manages manipulating the evaluator's
+  /// dependency source stack. It is specialized to be zero-cost for
+  /// requests that are not dependency sources.
+  template <typename Request, typename = detail::void_t<>>
+  struct IncrementalDependencyStackRAII {
+    IncrementalDependencyStackRAII(Evaluator &E, const Request &Req) {}
+  };
+
+  template <typename Request>
+  struct IncrementalDependencyStackRAII<
+      Request, typename std::enable_if<Request::isDependencySource>::type> {
+    NullablePtr<Evaluator> Eval;
+    IncrementalDependencyStackRAII(Evaluator &E, const Request &Req) {
+      auto Source = Req.readDependencySource(E);
+      // If there is no source to introduce, bail. This can occur if
+      // a request originates in the context of a module.
+      if (!Source.getPointer()) {
+        return;
+      }
+      E.dependencySources.emplace_back(Source);
+      Eval = &E;
+    }
+
+    ~IncrementalDependencyStackRAII() {
+      if (Eval.isNonNull())
+        Eval.get()->dependencySources.pop_back();
+    }
+  };
+
   /// Retrieve the request function for the given zone and request IDs.
   AbstractRequestFunction *getAbstractRequestFunction(uint8_t zoneID,
                                                       uint8_t requestID) const;
@@ -264,6 +303,7 @@ class Evaluator {
            typename std::enable_if<Request::isEverCached>::type * = nullptr>
   llvm::Expected<typename Request::OutputType>
   operator()(const Request &request) {
+    IncrementalDependencyStackRAII<Request> incDeps{*this, request};
     // The request can be cached, but check a predicate to determine
     // whether this particular instance is cached. This allows more
     // fine-grained control over which instances get cache.
@@ -279,6 +319,7 @@ class Evaluator {
            typename std::enable_if<!Request::isEverCached>::type * = nullptr>
   llvm::Expected<typename Request::OutputType>
   operator()(const Request &request) {
+    IncrementalDependencyStackRAII<Request> incDeps{*this, request};
     return getResultUncached(request);
   }
 
@@ -366,7 +407,9 @@ class Evaluator {
     FrontendStatsTracer statsTracer = make_tracer(stats, request);
     if (stats) reportEvaluatedRequest(*stats, request);
 
-    return getRequestFunction<Request>()(request, *this);
+    auto &&r = getRequestFunction<Request>()(request, *this);
+    reportEvaluatedResult<Request>(request, r);
+    return std::move(r);
   }
 
   /// Get the result of a request, consulting an external cache
@@ -377,8 +420,10 @@ class Evaluator {
   llvm::Expected<typename Request::OutputType>
   getResultCached(const Request &request) {
     // If there is a cached result, return it.
-    if (auto cached = request.getCachedResult())
+    if (auto cached = request.getCachedResult()) {
+      reportEvaluatedResult<Request>(request, *cached);
       return *cached;
+    }
 
     // Compute the result.
     auto result = getResultUncached(request);
@@ -403,7 +448,9 @@ class Evaluator {
     // If we already have an entry for this request in the cache, return it.
     auto known = cache.find_as(request);
     if (known != cache.end()) {
-      return known->second.template castTo<typename Request::OutputType>();
+      auto r = known->second.template castTo<typename Request::OutputType>();
+      reportEvaluatedResult<Request>(request, r);
+      return r;
     }
 
     // Compute the result.
@@ -416,6 +463,62 @@ class Evaluator {
     return result;
   }
 
+private:
+  // Report the result of evaluating a request that is not a dependency sink -
+  // which is to say do nothing.
+  template <typename Request,
+            typename std::enable_if<!Request::isDependencySink>::type * = nullptr>
+  void reportEvaluatedResult(const Request &r,
+                             const typename Request::OutputType &o) {}
+
+  // Report the result of evaluating a request that is a dependency sink.
+  template <typename Request,
+            typename std::enable_if<Request::isDependencySink>::type * = nullptr>
+  void reportEvaluatedResult(const Request &r,
+                             const typename Request::OutputType &o) {
+    if (auto *tracker = getActiveDependencyTracker())
+      r.writeDependencySink(*this, *tracker, o);
+  }
+
+  /// If there is an active dependency source, returns its
+  /// \c ReferencedNameTracker. Else, returns \c nullptr.
+  ReferencedNameTracker *getActiveDependencyTracker() const {
+    if (auto *source = getActiveDependencySourceOrNull())
+      return source->getRequestBasedReferencedNameTracker();
+    return nullptr;
+  }
+
+public:
+  /// Returns \c true if the scope of the current active source cascades.
+  ///
+  /// If there is no active scope, the result always cascades.
+  bool isActiveSourceCascading() const {
+    return getActiveSourceScope() == evaluator::DependencyScope::Cascading;
+  }
+
+  /// Returns the scope of the current active scope.
+  ///
+  /// If there is no active scope, the result always cascades.
+  evaluator::DependencyScope getActiveSourceScope() const {
+    if (dependencySources.empty()) {
+      return evaluator::DependencyScope::Cascading;
+    }
+    return dependencySources.back().getInt();
+  }
+
+  /// Returns the active dependency's source file, or \c nullptr if no
+  /// dependency source is active.
+  ///
+  /// The use of this accessor is strongly discouraged, as it implies that a
+  /// dependency sink is seeking to filter out names based on the files they
+  /// come from. Existing callers are being migrated to more reasonable ways
+  /// of judging the relevancy of a dependency.
+  SourceFile *getActiveDependencySourceOrNull() const {
+    if (dependencySources.empty())
+      return nullptr;
+    return dependencySources.back().getPointer();
+  }
+
 public:
   /// Print the dependencies of the given request as a tree.
   ///

include/swift/AST/EvaluatorDependencies.h

@@ -0,0 +1,106 @@
+//===--- EvaluatorDependencies.h - Auto-Incremental Dependencies -*- C++ -*-===//
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2020 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines data structures to support the request evaluator's
+// automatic incremental dependency tracking functionality.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SWIFT_AST_EVALUATOR_DEPENDENCIES_H
+#define SWIFT_AST_EVALUATOR_DEPENDENCIES_H
+
+#include "swift/AST/AttrKind.h"
+#include "swift/AST/SourceFile.h"
+#include "llvm/ADT/PointerIntPair.h"
+
+namespace swift {
+
+namespace evaluator {
+
+/// The "scope" of a dependency edge tracked by the evaluator.
+///
+/// Dependency scopes come in two flavors: cascading and private. A private
+/// edge captures dependencies discovered in contexts that are not visible to
+/// to other files. For example, a conformance to a private protocol, or the use
+/// of any names inside of a function body. A cascading edge, by contrast,
+/// captures dependencies discovered in the remaining visible contexts. These
+/// are types with at least \c internal visibility, names defined or used
+/// outside of function bodies with at least \c internal visibility, etc. A
+/// dependency that has cascading scope is so-named because upon traversing the
+/// edge, a reader such as the driver should continue transitively evaluating
+/// further dependency edges.
+///
+/// A cascading edge is always conservatively correct to produce, but it comes
+/// at the cost of increased resources spent (and possibly even wasted!) during
+/// incremental compilation. A private edge, by contrast, is more efficient for
+/// incremental compilation but it is harder to safely use.
+///
+/// To ensure that these edges are registered consistently with the correct
+/// scopes, requests that act as the source of dependency edges are required
+/// to specify a \c DependencyScope under which all evaluated sub-requests will
+/// register their dependency edges. In this way, \c DependencyScope values
+/// form a stack-like structure and are pushed and popped by the evaluator
+/// during the course of request evaluation.
+///
+/// When determining the kind of scope a request should use, always err on the
+/// side of a cascading scope unless there is absolute proof any discovered
+/// dependencies will be private. Inner requests may also defensively choose to
+/// flip the dependency scope from private to cascading in the name of safety.
+enum class DependencyScope : bool {
+  Private = false,
+  Cascading = true,
+};
+
+/// Returns a \c DependencyScope appropriate for the given (formal) access level.
+///
+/// :warning: This function exists to bridge the old manual reference
+/// dependencies code to the new evaluator-based reference dependencies code.
+/// The manual code often made private/cascading scope judgements based on the
+/// access level of a declaration. While this makes some sense intuitively, it
+/// does not necessarily capture an accurate picture of where real incremental
+/// dependencies lie. For example, references to formally private types can
+/// "escape" to contexts that have no reference to the private name if, say,
+/// the layout of that private type is taken into consideration by
+/// SILGen or IRGen in a separate file that references the declaration
+/// transitively. However, due to the density of the current dependency
+/// graph, redundancy in registered dependency edges, and the liberal use of
+/// cascading edges, we may be saved from the worst consequences of this
+/// modelling choice.
+///
+/// The use of access-levels for dependency decisions is an anti-pattern that
+/// should be revisited once finer-grained dependencies are explored more
+/// thoroughly.
+inline DependencyScope getScopeForAccessLevel(AccessLevel l) {
+  switch (l) {
+  case AccessLevel::Private:
+  case AccessLevel::FilePrivate:
+    return DependencyScope::Private;
+  case AccessLevel::Internal:
+  case AccessLevel::Public:
+  case AccessLevel::Open:
+    return DependencyScope::Cascading;
+  }
+}
+
+// A \c DependencySource is currently defined to be a parent source file and
+// an associated dependency scope.
+//
+// The \c SourceFile instance is an artifact of the current dependency system,
+// and should be scrapped if possible. It currently encodes the idea that
+// edges in the incremental dependency graph invalidate entire files instead
+// of individual contexts.
+using DependencySource = llvm::PointerIntPair<SourceFile *, 1, DependencyScope>;
+} // end namespace evaluator
+
+} // end namespace swift
+
+#endif // SWIFT_AST_EVALUATOR_DEPENDENCIES_H