First version of capture checker.

A squashed version of the following commits:

Handle byname parameters
Don't force symbol completion when printing flags or annotations
Check overrides and disallow non-local inferred capture types
Handle `this` in capture sets
Print capture variable dependencies under -Ydebug-cc
Avoid spurious error message

Avoid spurious error message
"cannot be tracked since its capture set is empty".

This arose in lazyref.scala for a DependentTypeTree in an anaonymois function.
Dependent type trees map to normal TypeTrees, not InferredTypeTrees (and things
go wrong if we try to change that).
Drop TopType

Consider bounds of type variables to be boxed
More tests
Avoid multiple maps when creating symbol infos

Use a single BiTypeMap to map from inferred result and parameters
to method info.

This improves efficiency and debuggability by reducing the frequence of
multiple stacked maps capture sets.
Refactor with CompareResult#andAlso
Refactoring: use isOK on CompareResult
Reflect inferred parameter types in enclosing method type

The variables in the inferred parameter type of an anonymous function need
to also show up in the closure type itself, so that they can be constrained.
Don't interpolate parameters of anonymous functions

Here, we should wait until we get the info from the outside, which can
be arbitrarily much later.
Compute upper approximation of bimapped sets from both sides
Fail when trying to add new elements to mapped sets

It's the safe option.
Print full origin trail of derived capture sets under -Ycc-debug
Fix isEmpty condition in well-formedness check
Make printing capture sets dependent on -Ycc-debug
Recursion brake for upperApprox
Fixes to upperApprox
Make instantiteRT a BiTypeMap

Otherwise we will not be able to do upper approximations of parameters.
Interpolate only variables at negative polarity

Interpolating covariant variables risks restricting capture sets to early.
For instance, when a variable has the capture set of a called function in
its capture set. When we have indirectly recursive calls it could be that
the capture set of a called function is not yet fully formed.
Interpolate type variables when symbols are completed
Allow for possibility that variables are constant
Only recomplete symbols if their info changes
Add completions to Rechecker

Complete val and def definitions lazily on first access. Now,
recheckDefDef and recheckValDef are called the first time the
new info of the defined symbol is needed, or, if the info is
never needed, when the typer gets to the definitions. This
only applied to definitions with inferred types. The others
are handled in typer sequence, as before.

The motivation of the change is that some modifications to
inferred types of symbols can be made in subclasses without
running into ordering problems.
More fixes for subCapture
New setting -Ycc-debug for more info on capture variables
Fix subCapture in frozen state

Previously, we still OKed two empty variables to be compared with
subcapture in the frozen state. This should give an error.
Direct comparisons of dependent function types
Revert: Special treatment of dependent functions in TypeComparer
change test
Also treat explicit capturing type arguments as boxed
Print subcapturing steps in -explain traces
Don't decorate type variables with additional capture sets
Boxed CapturingTypes
Drop unsound capture suppression if expected type is boxed

If expected type is boxed, the expression still contributes to the captured
variables of its environment.
Re-infer result types of anonymous functions
Keep erased implicit args
Special treatment of dependent functions in TypeComparer
Fix addFunctionRefinements
Always print refined function types as dependent functions.

Makes it easier to see what goes on.
Make CaptureSet ++ and ** simplify more
Refine function types when reinferring so that they can be dependent
Fix avoidance problem when typing blocks

We should not pass en expected type when rechecking the expression
of a block since that can add local references to global capture set variables.

Also: tests for lists and pairs
Print empty variables with "?"
Fix printing untyped annotations
Fix printing annotations in trees
Drop redundant code
Refactor map operations on capture sets
Intoduce Bi-Mapped CaptureSets

Report an error is a simply mapped capture set gets new elements that do not come
from the original souurce. Introduce a new abstraction of bi-mapped sets that accept
new elements and propagate them to the original source.
Add map operation to SimpleIdentitySet
Restrict tracked class parameters to vals
Handle local classes and secondary constructors
Fix CapturingType precedence when printing
First stab at handling classes
Bug fixes

 1. Fix canBeTracked for TermRefs

    only TermRefs where prefix is NoPrefix or `this` can be tracked. The
    others have to be widened.

 2. Fix rule for comparing capture refs on the left

 3. Be more careful where comparisons are frozen
Capture checker for functions

Author: odersky

compiler/src/dotty/tools/dotc/Compiler.scala

@@ -4,6 +4,7 @@ package dotc
 import core._
 import Contexts._
 import typer.{TyperPhase, RefChecks}
+import cc.CheckCaptures
 import parsing.Parser
 import Phases.Phase
 import transform._
@@ -81,6 +82,8 @@ class Compiler {
          new SpecializeApplyMethods, // Adds specialized methods to FunctionN
          new TryCatchPatterns,       // Compile cases in try/catch
          new PatternMatcher) ::      // Compile pattern matches
+    List(new PreRecheck) ::          // Preparations for check captures phase, enabled under -Ycc
+    List(new CheckCaptures) ::       // Check captures, enabled under -Ycc
     List(new ElimOpaque,             // Turn opaque into normal aliases
          new sjs.ExplicitJSClasses,  // Make all JS classes explicit (Scala.js only)
          new ExplicitOuter,          // Add accessors to outer classes from nested ones.
@@ -103,8 +106,6 @@ class Compiler {
          new TupleOptimizations,     // Optimize generic operations on tuples
          new LetOverApply,           // Lift blocks from receivers of applications
          new ArrayConstructors) ::   // Intercept creation of (non-generic) arrays and intrinsify.
-    List(new PreRecheck) ::          // Preparations for recheck phase, enabled under -Yrecheck
-    List(new TestRecheck) ::         // Test rechecking, enabled under -Yrecheck
     List(new Erasure) ::             // Rewrite types to JVM model, erasing all type parameters, abstract types and refinements.
     List(new ElimErasedValueType,    // Expand erased value types to their underlying implmementation types
          new PureStats,              // Remove pure stats from blocks

compiler/src/dotty/tools/dotc/Run.scala

@@ -19,7 +19,6 @@ import reporting.{Suppression, Action}
 import reporting.Diagnostic
 import reporting.Diagnostic.Warning
 import rewrites.Rewrites
-
 import profile.Profiler
 import printing.XprintMode
 import typer.ImplicitRunInfo
@@ -286,7 +285,7 @@ class Run(comp: Compiler, ictx: Context) extends ImplicitRunInfo with Constraint
     val fusedPhase = ctx.base.fusedContaining(prevPhase)
     val echoHeader = f"[[syntax trees at end of $fusedPhase%25s]] // ${unit.source}"
     val tree = if ctx.isAfterTyper then unit.tpdTree else unit.untpdTree
-    val treeString = tree.show(using ctx.withProperty(XprintMode, Some(())))
+    val treeString = fusedPhase.show(tree)
 
     last match {
       case SomePrintedTree(phase, lastTreeString) if lastTreeString == treeString =>

compiler/src/dotty/tools/dotc/ast/Desugar.scala

@@ -1758,6 +1758,9 @@ object desugar {
         flatTree(pats1 map (makePatDef(tree, mods, _, rhs)))
       case ext: ExtMethods =>
         Block(List(ext), Literal(Constant(())).withSpan(ext.span))
+      case CapturingTypeTree(refs, parent) =>
+        val annot = New(scalaDot(tpnme.retains), List(refs))
+        Annotated(parent, annot)
     }
     desugared.withSpan(tree.span)
   }
@@ -1896,6 +1899,8 @@ object desugar {
             case _ => traverseChildren(tree)
           }
         }.traverse(expr)
+      case CapturingTypeTree(refs, parent) =>
+        collect(parent)
       case _ =>
     }
     collect(tree)

compiler/src/dotty/tools/dotc/ast/Trees.scala

@@ -252,16 +252,10 @@ object Trees {
   /** Tree's denotation can be derived from its type */
   abstract class DenotingTree[-T >: Untyped](implicit @constructorOnly src: SourceFile) extends Tree[T] {
     type ThisTree[-T >: Untyped] <: DenotingTree[T]
-    override def denot(using Context): Denotation = typeOpt match {
+    override def denot(using Context): Denotation = typeOpt.stripped match
       case tpe: NamedType => tpe.denot
       case tpe: ThisType => tpe.cls.denot
-      case tpe: AnnotatedType => tpe.stripAnnots match {
-        case tpe: NamedType => tpe.denot
-        case tpe: ThisType => tpe.cls.denot
-        case _ => NoDenotation
-      }
       case _ => NoDenotation
-    }
   }
 
   /** Tree's denot/isType/isTerm properties come from a subtree

compiler/src/dotty/tools/dotc/ast/untpd.scala

@@ -146,6 +146,9 @@ object untpd extends Trees.Instance[Untyped] with UntypedTreeInfo {
     case Floating
   }
 
+  /** {x1, ..., xN} T   (only relevant under -Ycc) */
+  case class CapturingTypeTree(refs: List[Tree], parent: Tree)(implicit @constructorOnly src: SourceFile) extends TypTree
+
   /** Short-lived usage in typer, does not need copy/transform/fold infrastructure */
   case class DependentTypeTree(tp: List[Symbol] => Type)(implicit @constructorOnly src: SourceFile) extends Tree
 
@@ -649,6 +652,10 @@ object untpd extends Trees.Instance[Untyped] with UntypedTreeInfo {
       case tree: Number if (digits == tree.digits) && (kind == tree.kind) => tree
       case _ => finalize(tree, untpd.Number(digits, kind))
     }
+    def CapturingTypeTree(tree: Tree)(refs: List[Tree], parent: Tree)(using Context): Tree = tree match
+      case tree: CapturingTypeTree if (refs eq tree.refs) && (parent eq tree.parent) => tree
+      case _ => finalize(tree, untpd.CapturingTypeTree(refs, parent))
+
     def TypedSplice(tree: Tree)(splice: tpd.Tree)(using Context): ProxyTree = tree match {
       case tree: TypedSplice if splice `eq` tree.splice => tree
       case _ => finalize(tree, untpd.TypedSplice(splice)(using ctx))
@@ -714,6 +721,8 @@ object untpd extends Trees.Instance[Untyped] with UntypedTreeInfo {
         tree
       case MacroTree(expr) =>
         cpy.MacroTree(tree)(transform(expr))
+      case CapturingTypeTree(refs, parent) =>
+        cpy.CapturingTypeTree(tree)(transform(refs), transform(parent))
       case _ =>
         super.transformMoreCases(tree)
     }
@@ -775,6 +784,8 @@ object untpd extends Trees.Instance[Untyped] with UntypedTreeInfo {
         this(x, splice)
       case MacroTree(expr) =>
         this(x, expr)
+      case CapturingTypeTree(refs, parent) =>
+        this(this(x, refs), parent)
       case _ =>
         super.foldMoreCases(x, tree)
     }

compiler/src/dotty/tools/dotc/cc/CaptureAnnotation.scala

@@ -0,0 +1,63 @@
+package dotty.tools
+package dotc
+package cc
+
+import core.*
+import Types.*, Symbols.*, Contexts.*, Annotations.*
+import ast.Trees.*
+import ast.{tpd, untpd}
+import Decorators.*
+import config.Printers.capt
+import printing.Printer
+import printing.Texts.Text
+
+
+case class CaptureAnnotation(refs: CaptureSet, boxed: Boolean) extends Annotation:
+  import CaptureAnnotation.*
+  import tpd.*
+
+  override def tree(using Context) =
+    val elems = refs.elems.toList.map {
+      case cr: TermRef => ref(cr)
+      case cr: TermParamRef => untpd.Ident(cr.paramName).withType(cr)
+      case cr: ThisType => This(cr.cls)
+    }
+    val arg = repeated(elems, TypeTree(defn.AnyType))
+    New(symbol.typeRef, arg :: Nil)
+
+  override def symbol(using Context) = defn.RetainsAnnot
+
+  override def derivedAnnotation(tree: Tree)(using Context): Annotation =
+    unsupported("derivedAnnotation(Tree)")
+
+  def derivedAnnotation(refs: CaptureSet, boxed: Boolean)(using Context): Annotation =
+    if (this.refs eq refs) && (this.boxed == boxed) then this
+    else CaptureAnnotation(refs, boxed)
+
+  override def sameAnnotation(that: Annotation)(using Context): Boolean = that match
+    case CaptureAnnotation(refs2, boxed2) => refs == refs2 && boxed == boxed2
+    case _ => false
+
+  override def mapWith(tp: TypeMap)(using Context) =
+    val elems = refs.elems.toList
+    val elems1 = elems.mapConserve(tp)
+    if elems1 eq elems then this
+    else if elems1.forall(_.isInstanceOf[CaptureRef])
+    then derivedAnnotation(CaptureSet(elems1.asInstanceOf[List[CaptureRef]]*), boxed)
+    else EmptyAnnotation
+
+  override def refersToParamOf(tl: TermLambda)(using Context): Boolean =
+    refs.elems.exists {
+      case TermParamRef(tl1, _) => tl eq tl1
+      case _ => false
+    }
+
+  override def toText(printer: Printer): Text = refs.toText(printer)
+
+  override def hash: Int = (refs.hashCode << 1) | (if boxed then 1 else 0)
+
+  override def eql(that: Annotation) = that match
+    case that: CaptureAnnotation => (this.refs eq that.refs) && (this.boxed == boxed)
+    case _ => false
+
+end CaptureAnnotation

compiler/src/dotty/tools/dotc/cc/CaptureOps.scala

@@ -0,0 +1,82 @@
+package dotty.tools
+package dotc
+package cc
+
+import core.*
+import Types.*, Symbols.*, Contexts.*, Annotations.*
+import ast.{tpd, untpd}
+import Decorators.*
+import config.Printers.capt
+import util.Property.Key
+import tpd.*
+
+private val Captures: Key[CaptureSet] = Key()
+private val IsBoxed: Key[Unit] = Key()
+
+def retainedElems(tree: Tree)(using Context): List[Tree] = tree match
+  case Apply(_, Typed(SeqLiteral(elems, _), _) :: Nil) => elems
+  case _ => Nil
+
+extension (tree: Tree)
+
+  def toCaptureRef(using Context): CaptureRef = tree.tpe.asInstanceOf[CaptureRef]
+
+  def toCaptureSet(using Context): CaptureSet =
+    tree.getAttachment(Captures) match
+      case Some(refs) => refs
+      case None =>
+        val refs = CaptureSet(retainedElems(tree).map(_.toCaptureRef)*)
+          .showing(i"toCaptureSet $tree --> $result", capt)
+        tree.putAttachment(Captures, refs)
+        refs
+
+  def isBoxedCapturing(using Context): Boolean =
+    tree.hasAttachment(IsBoxed)
+
+  def setBoxedCapturing()(using Context): Unit =
+    tree.putAttachment(IsBoxed, ())
+
+extension (tp: Type)
+
+  def derivedCapturingType(parent: Type, refs: CaptureSet)(using Context): Type = tp match
+    case CapturingType(p, r, b) =>
+      if (parent eq p) && (refs eq r) then tp
+      else CapturingType(parent, refs, b)
+
+  /** If this is  type variable instantiated or upper bounded with a capturing type,
+   *  the capture set associated with that type. Extended to and-or types and
+   *  type proxies in the obvious way. If a term has a type with a boxed captureset,
+   *  that captureset counts towards the capture variables of the envirionment.
+   */
+  def boxedCaptured(using Context): CaptureSet =
+    def getBoxed(tp: Type): CaptureSet = tp match
+      case CapturingType(_, refs, boxed) => if boxed then refs else CaptureSet.empty
+      case tp: TypeProxy => getBoxed(tp.superType)
+      case tp: AndType => getBoxed(tp.tp1) ++ getBoxed(tp.tp2)
+      case tp: OrType => getBoxed(tp.tp1) ** getBoxed(tp.tp2)
+      case _ => CaptureSet.empty
+    getBoxed(tp)
+
+  def isBoxedCapturing(using Context) = !tp.boxedCaptured.isAlwaysEmpty
+
+  def canHaveInferredCapture(using Context): Boolean = tp match
+    case tp: TypeRef if tp.symbol.isClass =>
+      !tp.symbol.isValueClass && tp.symbol != defn.AnyClass
+    case _: TypeVar | _: TypeParamRef =>
+      false
+    case tp: TypeProxy =>
+      tp.superType.canHaveInferredCapture
+    case tp: AndType =>
+      tp.tp1.canHaveInferredCapture && tp.tp2.canHaveInferredCapture
+    case tp: OrType =>
+      tp.tp1.canHaveInferredCapture || tp.tp2.canHaveInferredCapture
+    case _ =>
+      false
+
+  def stripCapturing(using Context): Type = tp.dealiasKeepAnnots match
+    case CapturingType(parent, _, _) =>
+      parent.stripCapturing
+    case atd @ AnnotatedType(parent, annot) =>
+      atd.derivedAnnotatedType(parent.stripCapturing, annot)
+    case _ =>
+      tp