[move-only-addresses] Slicing some patches off of the larger patch.

docs/SIL.rst

@@ -2762,6 +2762,32 @@ sil-opt, one will see that we actually have an ownership violation due to the
 two uses of "value", one for initializing value2 and the other for the return
 value.
 
+Move Only Types
+---------------
+
+NOTE: This is experimental and is just an attempt to describe where the design
+is currently for others reading SIL today. It should not be interpreted as
+final.
+
+Currently there are two kinds of "move only types" in SIL: pure move only types
+that are always move only and move only wrapped types that are move only
+versions of copyable types. The invariant that values of Move Only type obey is
+that they can only be copied (e.x.: operand to a `copy_value`_, ``copy_addr [init]``) during the
+guaranteed passes when we are in Raw SIL. Once we are in non-Raw SIL though
+(i.e. Canonical and later SIL stages), a program is ill formed if one copies a
+move only type.
+
+The reason why we have this special rule for move only types is that this allows
+for SIL code generators to insert copies and then have a later guaranteed
+checker optimization pass recover the underlying move only semantics by
+reconstructing needed copies and removing unneeded copies using Ownership
+SSA. If any such copies are actually needed according to Ownership SSA, the
+checker pass emits a diagnostic stating that move semantics have been
+violated. If such a diagnostic is emitted then the checker pass transforms all
+copies on move only types to their explicit copy forms to ensure that once we
+leave the diagnostic passes and enter canonical SIL, our "copy" invariant is
+maintained.
+
 Runtime Failure
 ---------------
 
@@ -4219,6 +4245,25 @@ operations::
 If ``T`` is a trivial type, then ``copy_addr`` is always equivalent to its
 take-initialization form.
 
+It is illegal in non-Raw SIL to apply ``copy_addr [init]`` to a value that is
+move only.
+
+explicit_copy_addr
+``````````````````
+::
+
+  sil-instruction ::= 'explicit_copy_addr' '[take]'? sil-value
+                        'to' '[initialization]'? sil-operand
+
+  explicit_copy_addr [take] %0 to [initialization] %1 : $*T
+  // %0 and %1 must be of the same $*T address type
+
+This instruction is exactly the same as `copy_addr`_ except that it has special
+behavior for move only types. Specifically, an `explicit_copy_addr`_ is viewed
+as a copy_addr that is allowed on values that are move only. This is only used
+by a move checker after it has emitted an error diagnostic to preserve the
+general ``copy_addr [init]`` ban in Canonical SIL on move only types.
+
 destroy_addr
 ````````````
 ::
@@ -5557,6 +5602,8 @@ independent of the operand. In terms of specific types:
 In ownership qualified functions, a ``copy_value`` produces a +1 value that must
 be consumed at most once along any path through the program.
 
+It is illegal in non-Raw SIL to `copy_value`_ a value that is "move only".
+
 explicit_copy_value
 ```````````````````
 
@@ -5566,27 +5613,18 @@ explicit_copy_value
 
    %1 = explicit_copy_value %0 : $A
 
-Performs a copy of a loadable value as if by the value's type lowering and
-returns the copy. The returned copy semantically is a value that is completely
-independent of the operand. In terms of specific types:
-
-1. For trivial types, this is equivalent to just propagating through the trivial
-   value.
-2. For reference types, this is equivalent to performing a ``strong_retain``
-   operation and returning the reference.
-3. For ``@unowned`` types, this is equivalent to performing an
-   ``unowned_retain`` and returning the operand.
-4. For aggregate types, this is equivalent to recursively performing a
-   ``copy_value`` on its components, forming a new aggregate from the copied
-   components, and then returning the new aggregate.
-
-In ownership qualified functions, a ``explicit_copy_value`` produces a +1 value
-that must be consumed at most once along any path through the program.
-
-When move only variable checking is performed, ``explicit_copy_value`` is
+This is exactly the same instruction semantically as `copy_value`_ with the
+exception that when move only checking is performed, `explicit_copy_value`_ is
 treated as an explicit copy asked for by the user that should not be rewritten
 and should be treated as a non-consuming use.
 
+This is used for two things:
+
+1. Implementing a copy builtin for no implicit copy types.
+2. To enable the move checker, once it has emitted an error diagnostic, to still
+   produce valid Ownership SSA SIL at the end of the guaranteed optimization
+   pipeline when we enter the Canonical SIL stage.
+
 move_value
 ``````````
 

include/swift/SIL/SILBuilder.h

@@ -1037,6 +1037,14 @@ class SILBuilder {
         getSILDebugLocation(Loc), srcAddr, destAddr, isTake, isInitialize));
   }
 
+  ExplicitCopyAddrInst *
+  createExplicitCopyAddr(SILLocation Loc, SILValue srcAddr, SILValue destAddr,
+                         IsTake_t isTake, IsInitialization_t isInitialize) {
+    assert(srcAddr->getType() == destAddr->getType());
+    return insert(new (getModule()) ExplicitCopyAddrInst(
+        getSILDebugLocation(Loc), srcAddr, destAddr, isTake, isInitialize));
+  }
+
   BindMemoryInst *createBindMemory(SILLocation Loc, SILValue base,
                                    SILValue index, SILType boundType) {
     return insert(BindMemoryInst::create(getSILDebugLocation(Loc), base, index,

include/swift/SIL/SILCloner.h

@@ -1383,6 +1383,17 @@ SILCloner<ImplClass>::visitCopyAddrInst(CopyAddrInst *Inst) {
                 Inst->isInitializationOfDest()));
 }
 
+template <typename ImplClass>
+void SILCloner<ImplClass>::visitExplicitCopyAddrInst(
+    ExplicitCopyAddrInst *Inst) {
+  getBuilder().setCurrentDebugScope(getOpScope(Inst->getDebugScope()));
+  recordClonedInstruction(
+      Inst, getBuilder().createExplicitCopyAddr(
+                getOpLocation(Inst->getLoc()), getOpValue(Inst->getSrc()),
+                getOpValue(Inst->getDest()), Inst->isTakeOfSrc(),
+                Inst->isInitializationOfDest()));
+}
+
 template <typename ImplClass>
 void SILCloner<ImplClass>::visitMarkUnresolvedMoveAddrInst(
     MarkUnresolvedMoveAddrInst *Inst) {

include/swift/SIL/SILInstruction.h

@@ -5000,6 +5000,48 @@ class CopyAddrInst
   MutableArrayRef<Operand> getAllOperands() { return Operands.asArray(); }
 };
 
+/// ExplicitCopyAddrInst - A copy_addr that should not be optimized and should
+/// be viewed
+class ExplicitCopyAddrInst
+    : public InstructionBase<SILInstructionKind::ExplicitCopyAddrInst,
+                             NonValueInstruction>,
+      public CopyLikeInstruction {
+  friend SILBuilder;
+
+private:
+  FixedOperandList<2> Operands;
+  USE_SHARED_UINT8;
+
+  ExplicitCopyAddrInst(SILDebugLocation DebugLoc, SILValue Src, SILValue Dest,
+                       IsTake_t isTakeOfSrc,
+                       IsInitialization_t isInitializationOfDest);
+
+public:
+  SILValue getSrc() const { return Operands[Src].get(); }
+  SILValue getDest() const { return Operands[Dest].get(); }
+
+  void setSrc(SILValue V) { Operands[Src].set(V); }
+  void setDest(SILValue V) { Operands[Dest].set(V); }
+
+  IsTake_t isTakeOfSrc() const {
+    return IsTake_t(sharedUInt8().ExplicitCopyAddrInst.isTakeOfSrc);
+  }
+  IsInitialization_t isInitializationOfDest() const {
+    return IsInitialization_t(
+        sharedUInt8().ExplicitCopyAddrInst.isInitializationOfDest);
+  }
+
+  void setIsTakeOfSrc(IsTake_t T) {
+    sharedUInt8().ExplicitCopyAddrInst.isTakeOfSrc = (bool)T;
+  }
+  void setIsInitializationOfDest(IsInitialization_t I) {
+    sharedUInt8().ExplicitCopyAddrInst.isInitializationOfDest = (bool)I;
+  }
+
+  ArrayRef<Operand> getAllOperands() const { return Operands.asArray(); }
+  MutableArrayRef<Operand> getAllOperands() { return Operands.asArray(); }
+};
+
 /// "%token = bind_memory %0 : $Builtin.RawPointer, %1 : $Builtin.Word to $T"
 ///
 /// Binds memory at the raw pointer %0 to type $T with enough capacity

include/swift/SIL/SILNode.h

@@ -219,6 +219,10 @@ class alignas(8) SILNode :
       isTakeOfSrc : 1,
       isInitializationOfDest : 1);
 
+    SHARED_FIELD(ExplicitCopyAddrInst, uint8_t
+      isTakeOfSrc : 1,
+      isInitializationOfDest : 1);
+
     SHARED_FIELD(PointerToAddressInst, uint8_t
       isStrict : 1,
       isInvariant : 1);

include/swift/SIL/SILNodes.def

@@ -781,6 +781,8 @@ BRIDGED_NON_VALUE_INST(DebugValueInst, debug_value,
 #include "swift/AST/ReferenceStorage.def"
 BRIDGED_NON_VALUE_INST(CopyAddrInst, copy_addr,
                SILInstruction, MayHaveSideEffects, MayRelease)
+BRIDGED_NON_VALUE_INST(ExplicitCopyAddrInst, explicit_copy_addr,
+               SILInstruction, MayHaveSideEffects, MayRelease)
 BRIDGED_NON_VALUE_INST(DestroyAddrInst, destroy_addr,
                SILInstruction, MayHaveSideEffects, MayRelease)
 NON_VALUE_INST(EndLifetimeInst, end_lifetime,

lib/IRGen/IRGenSIL.cpp

@@ -1310,6 +1310,7 @@ class IRGenSILFunction :
   void visitDeallocPartialRefInst(DeallocPartialRefInst *i);
 
   void visitCopyAddrInst(CopyAddrInst *i);
+  void visitExplicitCopyAddrInst(ExplicitCopyAddrInst *i);
   void visitMarkUnresolvedMoveAddrInst(MarkUnresolvedMoveAddrInst *mai) {
     llvm_unreachable("Valid only when ownership is enabled");
   }
@@ -6884,6 +6885,30 @@ void IRGenSILFunction::visitCopyAddrInst(swift::CopyAddrInst *i) {
   }
 }
 
+void IRGenSILFunction::visitExplicitCopyAddrInst(
+    swift::ExplicitCopyAddrInst *i) {
+  SILType addrTy = i->getSrc()->getType();
+  const TypeInfo &addrTI = getTypeInfo(addrTy);
+  Address src = getLoweredAddress(i->getSrc());
+  // See whether we have a deferred fixed-size buffer initialization.
+  auto &loweredDest = getLoweredValue(i->getDest());
+  assert(!loweredDest.isUnallocatedAddressInBuffer());
+  Address dest = loweredDest.getAnyAddress();
+  if (i->isInitializationOfDest()) {
+    if (i->isTakeOfSrc()) {
+      addrTI.initializeWithTake(*this, dest, src, addrTy, false);
+    } else {
+      addrTI.initializeWithCopy(*this, dest, src, addrTy, false);
+    }
+  } else {
+    if (i->isTakeOfSrc()) {
+      addrTI.assignWithTake(*this, dest, src, addrTy, false);
+    } else {
+      addrTI.assignWithCopy(*this, dest, src, addrTy, false);
+    }
+  }
+}
+
 // bind_memory and rebind_memory are no-ops because Swift TBAA info is not
 // lowered to LLVM IR TBAA, and the output token is ignored except for
 // verification.

lib/SIL/IR/OperandOwnership.cpp

@@ -147,6 +147,7 @@ OPERAND_OWNERSHIP(TrivialUse, CheckedCastAddrBranch)
 OPERAND_OWNERSHIP(TrivialUse, CondBranch)
 OPERAND_OWNERSHIP(TrivialUse, CondFail)
 OPERAND_OWNERSHIP(TrivialUse, CopyAddr)
+OPERAND_OWNERSHIP(TrivialUse, ExplicitCopyAddr)
 OPERAND_OWNERSHIP(TrivialUse, MarkUnresolvedMoveAddr)
 OPERAND_OWNERSHIP(TrivialUse, DeallocStack)
 OPERAND_OWNERSHIP(TrivialUse, DeinitExistentialAddr)

lib/SIL/IR/SILInstructions.cpp

@@ -1172,6 +1172,15 @@ CopyAddrInst::CopyAddrInst(SILDebugLocation Loc, SILValue SrcLValue,
       bool(isInitializationOfDest);
   }
 
+  ExplicitCopyAddrInst::ExplicitCopyAddrInst(
+      SILDebugLocation Loc, SILValue SrcLValue, SILValue DestLValue,
+      IsTake_t isTakeOfSrc, IsInitialization_t isInitializationOfDest)
+      : InstructionBase(Loc), Operands(this, SrcLValue, DestLValue) {
+    sharedUInt8().ExplicitCopyAddrInst.isTakeOfSrc = bool(isTakeOfSrc);
+    sharedUInt8().ExplicitCopyAddrInst.isInitializationOfDest =
+        bool(isInitializationOfDest);
+  }
+
 BindMemoryInst *
 BindMemoryInst::create(SILDebugLocation Loc, SILValue Base, SILValue Index,
                        SILType BoundType, SILFunction &F) {

lib/SIL/IR/SILPrinter.cpp

@@ -1749,6 +1749,15 @@ class SILPrinter : public SILInstructionVisitor<SILPrinter> {
     *this << getIDAndType(CI->getDest());
   }
 
+  void visitExplicitCopyAddrInst(ExplicitCopyAddrInst *CI) {
+    if (CI->isTakeOfSrc())
+      *this << "[take] ";
+    *this << Ctx.getID(CI->getSrc()) << " to ";
+    if (CI->isInitializationOfDest())
+      *this << "[initialization] ";
+    *this << getIDAndType(CI->getDest());
+  }
+
   void visitMarkUnresolvedMoveAddrInst(MarkUnresolvedMoveAddrInst *CI) {
     *this << Ctx.getID(CI->getSrc()) << " to ";
     *this << getIDAndType(CI->getDest());

lib/SIL/Parser/ParseSIL.cpp

@@ -4920,6 +4920,37 @@ bool SILParser::parseSpecificSILInstruction(SILBuilder &B,
                                    IsInitialization_t(IsInit));
       break;
     }
+    case SILInstructionKind::ExplicitCopyAddrInst: {
+      bool IsTake = false, IsInit = false;
+      UnresolvedValueName SrcLName;
+      SILValue DestLVal;
+      SourceLoc ToLoc, DestLoc;
+      Identifier ToToken;
+      if (parseSILOptional(IsTake, *this, "take") || parseValueName(SrcLName) ||
+          parseSILIdentifier(ToToken, ToLoc, diag::expected_tok_in_sil_instr,
+                             "to") ||
+          parseSILOptional(IsInit, *this, "initialization") ||
+          parseTypedValueRef(DestLVal, DestLoc, B) ||
+          parseSILDebugLocation(InstLoc, B))
+        return true;
+
+      if (ToToken.str() != "to") {
+        P.diagnose(ToLoc, diag::expected_tok_in_sil_instr, "to");
+        return true;
+      }
+
+      if (!DestLVal->getType().isAddress()) {
+        P.diagnose(DestLoc, diag::sil_invalid_instr_operands);
+        return true;
+      }
+
+      SILValue SrcLVal =
+          getLocalValue(SrcLName, DestLVal->getType(), InstLoc, B);
+      ResultVal =
+          B.createExplicitCopyAddr(InstLoc, SrcLVal, DestLVal, IsTake_t(IsTake),
+                                   IsInitialization_t(IsInit));
+      break;
+    }
     case SILInstructionKind::MarkUnresolvedMoveAddrInst: {
       UnresolvedValueName SrcLName;
       SILValue DestLVal;

lib/SIL/Utils/InstructionUtils.cpp

@@ -630,6 +630,15 @@ RuntimeEffect swift::getRuntimeEffect(SILInstruction *inst, SILType &impactType)
       return RuntimeEffect::MetaData | RuntimeEffect::RefCounting;
     return RuntimeEffect::MetaData;
   }
+  case SILInstructionKind::ExplicitCopyAddrInst: {
+    auto *ca = cast<ExplicitCopyAddrInst>(inst);
+    impactType = ca->getSrc()->getType();
+    if (!ca->isInitializationOfDest())
+      return RuntimeEffect::MetaData | RuntimeEffect::Releasing;
+    if (!ca->isTakeOfSrc())
+      return RuntimeEffect::MetaData | RuntimeEffect::RefCounting;
+    return RuntimeEffect::MetaData;
+  }
   // Equivalent to a copy_addr [init]
   case SILInstructionKind::MarkUnresolvedMoveAddrInst: {
     return RuntimeEffect::MetaData | RuntimeEffect::RefCounting;

lib/SIL/Verifier/SILVerifier.cpp

@@ -2189,6 +2189,10 @@ class SILVerifier : public SILVerifierBase<SILVerifier> {
               "Load with unqualified ownership in a qualified function");
       break;
     case LoadOwnershipQualifier::Copy:
+      require(LI->getModule().getStage() == SILStage::Raw ||
+                  !LI->getOperand()->getType().isMoveOnly(),
+              "'MoveOnly' types can only be copied in Raw SIL?!");
+      [[fallthrough]];
     case LoadOwnershipQualifier::Take:
       require(F.hasOwnership(),
               "Load with qualified ownership in an unqualified function");
@@ -2655,15 +2659,17 @@ class SILVerifier : public SILVerifierBase<SILVerifier> {
       require(Elt->getType().isAddress(), "MFE must refer to variable addrs");
   }
 
-  void checkCopyAddrInst(CopyAddrInst *SI) {
-    require(SI->getSrc()->getType().isAddress(),
-            "Src value should be lvalue");
-    require(SI->getDest()->getType().isAddress(),
+  void checkCopyAddrInst(CopyAddrInst *cai) {
+    require(cai->getSrc()->getType().isAddress(), "Src value should be lvalue");
+    require(cai->getDest()->getType().isAddress(),
             "Dest address should be lvalue");
-    requireSameType(SI->getDest()->getType(), SI->getSrc()->getType(),
+    requireSameType(cai->getDest()->getType(), cai->getSrc()->getType(),
                     "Store operand type and dest type mismatch");
-    require(F.isTypeABIAccessible(SI->getDest()->getType()),
+    require(F.isTypeABIAccessible(cai->getDest()->getType()),
             "cannot directly copy type with inaccessible ABI");
+    require(cai->getModule().getStage() == SILStage::Raw ||
+                (cai->isTakeOfSrc() || !cai->getSrc()->getType().isMoveOnly()),
+            "'MoveOnly' types can only be copied in Raw SIL?!");
   }
 
   void checkMarkUnresolvedMoveAddrInst(MarkUnresolvedMoveAddrInst *SI) {

lib/SILOptimizer/UtilityPasses/SerializeSILPass.cpp

@@ -314,6 +314,7 @@ static bool hasOpaqueArchetype(TypeExpansionContext context,
   case SILInstructionKind::Store##Name##Inst:
 #include "swift/AST/ReferenceStorage.def"
   case SILInstructionKind::CopyAddrInst:
+  case SILInstructionKind::ExplicitCopyAddrInst:
   case SILInstructionKind::MarkUnresolvedMoveAddrInst:
   case SILInstructionKind::DestroyAddrInst:
   case SILInstructionKind::EndLifetimeInst:

lib/SILOptimizer/Utils/SILInliner.cpp

@@ -884,6 +884,7 @@ InlineCost swift::instructionInlineCost(SILInstruction &I) {
   case SILInstructionKind::CopyBlockInst:
   case SILInstructionKind::CopyBlockWithoutEscapingInst:
   case SILInstructionKind::CopyAddrInst:
+  case SILInstructionKind::ExplicitCopyAddrInst:
   case SILInstructionKind::MarkUnresolvedMoveAddrInst:
   case SILInstructionKind::RetainValueInst:
   case SILInstructionKind::RetainValueAddrInst: