[MLIR][ExecutionEngine] Introduce shared library

[Dinistro]

This commit introduces a shared library for the MLIR execution engine. This library is only built when LLVM_BUILD_LLVM_DYLIB is set. Having such a library allows downstream users to depend on the execution engine without giving up dynamic linkage. This is especially important for CPU runner-style tools, as they link against large parts of MLIR and LLVM.

It is alternatively possible to modify the MLIRExecutionEngine target when LLVM_BUILD_LLVM_DYLIB is set, to avoid duplicated libraries.

[llvmbot]

@llvm/pr-subscribers-mlir-execution-engine

@llvm/pr-subscribers-mlir

Author: Christian Ulmann (Dinistro)

Changes

This commit introduces a shared library for the MLIR execution engine. This library is only built when LLVM_BUILD_LLVM_DYLIB is set. Having such a library allows downstream users to depend on the execution engine without giving up dynamic linkage. This is especially important for CPU runner-style tools, as they link against large parts of MLIR and LLVM.

It is alternatively possible to modify the MLIRExecutionEngine target when LLVM_BUILD_LLVM_DYLIB is set, to avoid duplicated libraries.

---

Full diff: https://github.com/llvm/llvm-project/pull/87067.diff

1 Files Affected:

• (modified) mlir/lib/ExecutionEngine/CMakeLists.txt (+23)

diff --git a/mlir/lib/ExecutionEngine/CMakeLists.txt b/mlir/lib/ExecutionEngine/CMakeLists.txt
index b7e448d5417ea9..a091944b9ee7df 100644
--- a/mlir/lib/ExecutionEngine/CMakeLists.txt
+++ b/mlir/lib/ExecutionEngine/CMakeLists.txt
@@ -97,6 +97,29 @@ add_mlir_library(MLIRExecutionEngine
   MLIRTargetLLVMIRExport
   )
 
+if(LLVM_BUILD_LLVM_DYLIB)
+  # Build a shared library for the execution engine. Some downstream projects
+  # use this library to build their own CPU runners while preserving dynamic
+  # linkage.
+  add_mlir_library(MLIRExecutionEngineShared
+    ExecutionEngine.cpp
+    SHARED
+
+    EXCLUDE_FROM_LIBMLIR
+
+    ADDITIONAL_HEADER_DIRS
+    ${MLIR_MAIN_INCLUDE_DIR}/mlir/ExecutionEngine
+
+    # Ensures that all necessary dependencies are resolved.
+    DEPENDS
+    MLIRExecutionEngine
+
+    LINK_LIBS PUBLIC
+    LLVM
+    MLIR
+    )
+endif()
+
 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 add_mlir_library(MLIRJitRunner
   JitRunner.cpp

[makslevental]

I recently started building my mlir wheels with -DLLVM_BUILD_LLVM_DYLIB=ON and today I had a downstream python bindings test fail (on Mac):

Could not load /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libmlir_async_runtime.dylib:
tests/test_async.py 
  dlopen(/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libmlir_async_runtime.dylib, 0x0009): Library not loaded: '@loader_path/libLLVM.dylib'
  Referenced from: '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libmlir_async_runtime.dylib'
  Reason: tried: '/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/mlir/_mlir_libs/libLLVM.dylib' (no such file), '/usr/local/lib/libLLVM.dylib' (no such file), '/usr/lib/libLLVM.dylib' (no such file) 

Sanity checking

(numba-mlir) mlevental@maksims-mbp llvm-project % otool -L mlir/_mlir_libs/libmlir_async_runtime.dylib
mlir/_mlir_libs/libmlir_async_runtime.dylib:
        @rpath/libmlir_async_runtime.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/libLLVM.dylib (compatibility version 1.0.0, current version 19.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

so that's unfortunate. I'm guessing if I do a ldd libmlir_async_runtime.so on a linux box I'll see the same thing and the runtime paths are being handled more permissively (auditwheel/delocate-wheel does inject libLLVM.dylib into the package but maybe Mac doesn't consider where it's injected as part of the @loader_path :person_shrugging:). I'll double check that tomorrow.

Went and checked the CMake and found this PR. Note sure how MLIRExecutionEngineShared connects to libmlir_async_runtime. Anyway not sure what we should do - for the moment I'm going to patch out that target in my builds so it's not a big deal for me personally but nonetheless it seems like a bug.

Btw futzing with the paths (eg by moving libLLVM.dylib to a place where Mac runtime linker can find it) is a no-go because of some cl::opt collision:

test_async.py::test_simple_parfor 
Skipped: only check latest
LLVM ERROR: Option 'basic' already exists!
Fatal Python error: Aborted

[makslevental]

Just confirming that indeed on linux we see the same linking:

(mlir-aie) mlevental@mlevental-F7BSC:~/dev_projects/mlir-python-extras/mlir$ ldd mlir/_mlir_libs/libmlir_async_runtime.so
        linux-vdso.so.1 (0x00007ffebf759000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb11a72c000)
        libLLVM-4731abf8.so => /home/mlevental/miniconda3/envs/mlir-aie/lib/python3.11/site-packages/mlir/_mlir_libs/../../mlir_python_bindings.libs/libLLVM-4731abf8.so (0x00007fb115000000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb114c00000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb114f19000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb11a706000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb114800000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb11a763000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb11a701000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb11a6fc000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb11a6e0000)

[joker-eph]

Went and checked the CMake and found this PR. Note sure how MLIRExecutionEngineShared connects to libmlir_async_runtime

I'm not sure I follow either: why do think there is a relationship between your issue and this PR?

Just confirming that indeed on linux we see the same linking:

In this case it links to a particular version of libLLVM (libLLVM-4731abf8.so) and find a path to it, what do you mean by "the same linking"? Were you hoping that there wouldn't be any mention of libLLVM here?

[makslevental]

I'm not sure I follow either: why do think there is a relationship between your issue and this PR?

It was a hypothesis based on likely causes (libmlir_async_runtime.so is built in this CMake + this CMake recently introduced a connection to libLLVM). I have actually falsified the hypothesis just now but nonetheless that's the reason I came to this PR.

Do you know how else it would be possible libmlir_async_runtime could inherit a linking to libLLVM?

"the same linking"?

The same linking behavior (the hash isn't important) i.e., it's not only on Mac libmlir_async_runtime gets linked to libLLVM but also on Linux.

[joker-eph]

Do you know how else it would be possible libmlir_async_runtime could inherit a linking to libLLVM?

If I have to guess, I would think it comes from:

  # MLIR libraries uniformly depend on LLVMSupport.  Just specify it once here.
  list(APPEND ARG_LINK_COMPONENTS Support)
  _check_llvm_components_usage(${name} ${ARG_LINK_LIBS})

inside add_mlir_library, but that has been like this "forever".

[makslevental]

inside add_mlir_library, but that has been like this "forever".

So maybe this the correct answer but then I don't understand why eg some of the other libraries in this very same touched CMake don't inherit the link dep:

(mlir-aie) mlevental@mlevental-F7BSC:~/dev_projects/mlir-wheels$ ldd /home/mlevental/dev_projects/mlir-wheels/build/temp/lib/libmlir_arm_runner_utils.so
        linux-vdso.so.1 (0x00007fff65ca6000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5017e00000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5017a00000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5017d19000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f50180c5000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f501807c000)
(mlir-aie) mlevental@mlevental-F7BSC:~/dev_projects/mlir-wheels$ ldd /home/mlevental/dev_projects/mlir-wheels/build/temp/lib/libmlir_float16_utils.so
        linux-vdso.so.1 (0x00007fff82987000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f59a6c00000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f59a6800000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f59a6b19000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f59a6f72000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f59a6f29000)

Note, the answer isn't EXCLUDE_FROM_LIBMLIR because

  add_mlir_library(mlir_float16_utils
    SHARED
    Float16bits.cpp

    EXCLUDE_FROM_LIBMLIR
    )

but

  add_mlir_library(mlir_arm_runner_utils
    SHARED
    ArmRunnerUtils.cpp)

[makslevental]

Oh duh it's probably because they don't use anything from Support and thus the linker DCEs the link. Okay sorry for the noise - my mistake.

[stellaraccident]

Oh duh it's probably because they don't use anything from Support and thus the linker DCEs the link. Okay sorry for the noise - my mistake.

Yes, that is probably it.

In general, I would have thought that runtime support libraries like this should not have linked in llvm support at all, but that is an opinion I'm not going to fight for. There are a lot of problems with the things tied to ExecutionEngine and I take a live and let live attitude to it, while trying to not end up in too much of a support burden.

If the build was set up differently, there should be a static only support library variant built with hidden visibility. And that should be used for such cases. But that would be pretty hard to layer properly without breaking support out as its own thing and doing structural build upgrades on it.

[Zentrik]

Hi, this doesn't seem to build for me on windows using mingw. See #106859 for more details, thanks.