Give IRGenModule Ownership of its LLVMContext

[CodaFi]

Formalize the ownership contract between code generation, the LLVM Modules it generates, and the LLVMContext those modules are allocated in. Previously, the context to generate code in was under caller control. In practice, most callers passed around the global context. With David's patch, IRGenModule now allocates and destroys the context itself after code generation has completed. This was mostly harmless. Mostly.

The first place this disrupted was ORCJIT, which needed ownership of the context the generated module was allocated in. This necessitated the introduction of a new move-only type that manages the lifetime of the Module-Context pairs used by IRGenModule. I've called it swift::GeneratedModule. Its API contract is simple: The module and context live and die together, and ownership of a GeneratedModule is exclusive by construction. There are also a very limited set of ownership transfer operations on a GeneratedModule value. GeneratedModule::intoThreadSafeContext consumes a GeneratedModule and produces an orc::ThreadSafeModule suitable for use by the JIT. The remaining release operation mirrors the release operation on std::unique_ptr, except it consumes the GeneratedModule value in the process. This supports the second part of the story:

The second place this disrupted was the legacy integrated REPL. The REPL generates a fresh module for each "line", links the generated modules together, then passes them to its ExecutionEngine. The LLVM Linker has never handled linking modules allocated in separate contexts. Thus, the REPL used to just allocate everything into Swift's ad-hoc global context. Given that it is no longer under caller control, one solution would be to clone the module into the global context and link that. However, cloning LLVM Modules across contexts is also not a first-class operation. Instead, introduce an egregious hack that round-trips the module through one of LLVM's serialization paths. I have chosen the IR printer, but the bitcode printer would work equally well.

We need to delete the integrated REPL.

The third place this disrupted was LLDB. There, it seems like they just need to extract the module and context in order to form an IRExecutionUnit, so I've cleaned them up under that assumption. The Clang REPL code seems built in a very similar manner, which gives me some hope.

In theory, this patchset is NFC, but given the linker hack in the REPL - and that LLVM does not test round-tripping at a semantic level very well - I am not confident in that claim.

Even if we don't wind up going through with the parts of this patch that sequester the ownership of the LLVM Context, I can still break off the little cleanup pieces from this patchset - there's some const-qualification and other little mechanical improvements that would be nice to have.

[davidungar]

Only had time to give it a very quick look right away but it looks wonderful to me! Very nice to see it passes the consistency test. Way to go, @CodaFi !

[davidungar]

Only had time to give it a very quick look right away but it looks wonderful to me! Very nice to see it passes the consistency test. Way to go, @CodaFi !

[CodaFi]

swiftlang/llvm-project#1088

@swift-ci test

[CodaFi]

Screwed up LLDB. This first round is going to fail.

[CodaFi]

swiftlang/llvm-project#1088

@swift-ci test

[lhames]

This looks good to me. I like the clarity of ownership in GenerateModule.

Since re-using contexts came up on the REPL use case I'll just mention this: orc::ThreadSafeModule and orc::ThreadSafeContext were designed to account for this situation if you want to use them directly. They invert the usual Context/Module ownership relationship: Instead of Contexts uniquely owning multiple Modules, ThreadSafeModules share ownership of a ThreadSafeContext. That allows you to write APIs that optionally take a Context: If the client has one they want to use then they can (re)use it, if not the function can create a new one. E.g.

  ThreadSafeModule createModule(StringRef FileName,
                                Optional<ThreadSafeModule> Ctx = None) {
    // If the user didn't supply a context then create one.
    if (!Ctx)
      Ctx.emplace(std::make_unique<llvm::LLVMContext>());

    // Lock the ThreadSafeContext wrapper.
    auto ContextLock = Ctx->getLock();

    // Create the module.
    std::unique_ptr<llvm::Module> M = loadModuleOnContext(FileName, Ctx->getContext());

    // Wrap the Module in a ThreadSafeModule and return.
    // The return value assumes shared ownership of Ctx.
    return ThreadSafeModule(std::move(M), std::move(*Ctx));
  }

Though since the REPL is the only client that would benefit from this right now and you're thinking of removing that anyway the complexity may not be worth it.

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 03b19f3

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 03b19f3

[CodaFi]

swiftlang/llvm-project#1088

@swift-ci test

[CodaFi]

swiftlang/llvm-project#1088

@swift-ci asan test

[CodaFi]

swiftlang/llvm-project#1088

@swift-ci test source compatibility

[CodaFi]

⛵