[Distributed] Fix too restrictive distributed witness isolation checking

[ktoso]

Resolves rdar://94779780
Blocked by: #59356

---

Distributed witness checking is implemented in order to avoid "peeling off" the distributed-ness from a distributed actor.
We cannot allow upcasting a DA to some arbitrary protocol P, where that protocol P requirement isn't async + throws, because cross-actor calls must always have this effect.

We missed to handle an edge case in the model, which is requirements declared on a protocol DAP: DistributedActor which is where we CANNOT "peel off" the distributedness off the distributed actor, by upcasting it to that DAP.

protocol WatchingDA: DistributedActor {
  func terminated(da: String) async
}

distributed actor DA_WatchingDA: WatchingDA {
  func terminated(da: String) { } // should be fine
}

// The isolation checking itself is UNCHANGED and relies on the usual rules:
func test<WDA: WatchingDA>(da: WDA) async throws {
  try await da.terminated(da: "") // error: because of the normal distributed isolation checking rules; nothing new here
  await da.whenLocal { await $0.terminated(da: "OK") } // this is allowed
  // self.terminated(da:) is of course also allowed

  await da.whenLocal { $0.terminated(da: "OK") } // crashes because of  #59356
}

Errors forcing us to adopt nonisolated or distributed for this method; while it is strictly speaking not necessary.

We CAN witness this method, but it will be impossible to call when erased to WatchingDA because distributed actor isolation will prevent it from being called (!),
and this is exactly what we want. To only be able to call "inside" or "when known to be local", which is how frameworks can implement their lifecycle management hooks.

Example:

In practice this missing checking rule caused us to have to write such terrible workarounds:

    public nonisolated func terminated(actor id: ActorID) async { // HORRIBLE; we know we are the local actor here always when this is called (because OUTER caller does this whenLocal dance)
       await self.whenLocal { __secretlyKnownToBeLocal in
           await __secretlyKnownToBeLocal.probe.tell("Received terminated: \(id)")
        }
    }

     public distributed func terminated(actor id: ActorID) async { // REALLY NOT distributed
         self.probe.tell("Received terminated: \(id)")
     }

[ktoso]

I did tests with working around the limitation of actor isolation with generics -- once #59356 is done we can undo the hack here 👍

[ktoso]

@swift-ci please smoke test

[DougGregor]

Looks okay, but I think it can be simplified somewhat.

[DougGregor]

We can't have distributed requirements outside of DistributedActor-based protocols, right? So we don't need this isDistributedDecl check?

[DougGregor]

Or did you mean isDistributedDecl(witness)?

[ktoso]

I revisited this all, added more tests and simplified: 8e6d721

[ktoso]

Thank you! I'll dig into the comments 🙏

[ktoso]

@swift-ci please smoke test and merge