[openmp][WebAssembly] Allow openmp to compile and run under emscripten toolchain

[arsnyder16]

• Separate wasi and emscripten as they have different constraints and abilities
  • Emscripten mimics Linux/POSIX by statically linking the musl runtime. This allow nearly all KMP_OS_LINUX code paths to work correctly. There are only a few places that need to be adjusted related to dynamic linking (dl_open)
• Internally link openmp globals
  • With CommonLinkage it is needed to emit them in an assembly file, now they are defined and used within each compilation unit
  • With ExternalLinkage they suffer from duplicate symbols during linking for unnamed globals like reduction/critical
  • Interestingly this aligns with the TODO comment above this code

[abrown]

I haven't dug into this PR much yet review-wise but I think it is on the right track:

• I was shaky at best on the differences between InternalLinkage versus ExternalLinkage when I originally pieced [openmp][wasm] Allow compiling OpenMP to WebAssembly  #71297 together so this seems to make sense
• it is true that Emscripten and WASI have quite different paradigms so it is plausible that we need to handle the Emscripten bits differently

I tested this PR out in a simplistic WASI example and everything seemed to work as before on the WASI side. This "OpenMP + WebAssembly" support is still very experimental so I think that is probably the best we can do for now. Of course, it would be useful to have an Emscripten version of that around — @arsnyder16, do you have something like that?

[arsnyder16]

I haven't dug into this PR much yet review-wise but I think it is on the right track:

• I was shaky at best on the differences between InternalLinkage versus ExternalLinkage when I originally pieced [openmp][wasm] Allow compiling OpenMP to WebAssembly  #71297 together so this seems to make sense

• it is true that Emscripten and WASI have quite different paradigms so it is plausible that we need to handle the Emscripten bits differently

I tested this PR out in a simplistic WASI example and everything seemed to work as before on the WASI side. This "OpenMP + WebAssembly" support is still very experimental so I think that is probably the best we can do for now. Of course, it would be useful to have an Emscripten version of that around — @arsnyder16, do you have something like that?

Yea I can supply an emscripten variant, I'll fork your branch, I'll add a few more openmp scenarios as well to it to mimic the various link scenarios I encountered as well

[arsnyder16]

I haven't dug into this PR much yet review-wise but I think it is on the right track:

• I was shaky at best on the differences between InternalLinkage versus ExternalLinkage when I originally pieced [openmp][wasm] Allow compiling OpenMP to WebAssembly  #71297 together so this seems to make sense
• it is true that Emscripten and WASI have quite different paradigms so it is plausible that we need to handle the Emscripten bits differently

I tested this PR out in a simplistic WASI example and everything seemed to work as before on the WASI side. This "OpenMP + WebAssembly" support is still very experimental so I think that is probably the best we can do for now. Of course, it would be useful to have an Emscripten version of that around — @arsnyder16, do you have something like that?

abrown/wasm-openmp-examples#2

You can see a few of the issues here depending on the variants of clang and openmp, wasi vs emscripten as well.

For reference on the linkage, both wasi and emscripten suffer the same issues

ExternalLinkage

/root/wasm-openmp-examples/wasi-sdk-22.0/bin/clang -fopenmp=libomp -g --sysroot=/root/wasm-openmp-examples/wasi-sdk-22.0/share/wasi-sysroot --target=wasm32-wasi-threads \
  -I/root/wasm-openmp-examples/build/runtime/src -I/root/wasm-openmp-examples/wasi-sdk-22.0/share/wasi-sysroot/include -pthread \
  -Wl,--import-memory,--export-memory,--max-memory=67108864 example.c other.c \
  -L/root/wasm-openmp-examples/build/runtime/src -lomp \
  -lwasi-emulated-getpid \
  -o example.wasm
wasm-ld: error: duplicate symbol: .gomp_critical_user_.var
>>> defined in /tmp/example-5ca29e.o
>>> defined in /tmp/other-19151f.o

wasm-ld: error: duplicate symbol: .gomp_critical_user_NAME.var
>>> defined in /tmp/example-5ca29e.o
>>> defined in /tmp/other-19151f.o

wasm-ld: error: duplicate symbol: .gomp_critical_user_.reduction.var
>>> defined in /tmp/example-5ca29e.o
>>> defined in /tmp/other-19151f.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:31: example.wasm] Error 1

CommonLinkage

/root/wasm-openmp-examples/wasi-sdk-22.0/bin/clang -fopenmp=libomp -g --sysroot=/root/wasm-openmp-examples/wasi-sdk-22.0/share/wasi-sysroot --target=wasm32-wasi-threads \
  -I/root/wasm-openmp-examples/build/runtime/src -I/root/wasm-openmp-examples/wasi-sdk-22.0/share/wasi-sysroot/include -pthread \
  -Wl,--import-memory,--export-memory,--max-memory=67108864 example.c other.c \
  -L/root/wasm-openmp-examples/build/runtime/src -lomp \
  -lwasi-emulated-getpid \
  -o example.wasm
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_NAME.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_.reduction.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_.reduction.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_NAME.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_.reduction.var
wasm-ld: error: /tmp/example-1bd196.o: undefined symbol: .gomp_critical_user_.reduction.var

InternalLinkage

/root/wasm-openmp-examples/wasi-sdk-22.0/bin/clang -fopenmp=libomp -g --sysroot=/root/wasm-openmp-examples/wasi-sdk-22.0/share/wasi-sysroot --target=wasm32-wasi-threads \
  -I/root/wasm-openmp-examples/build/runtime/src -I/root/wasm-openmp-examples/wasi-sdk-22.0/share/wasi-sysroot/include -pthread \
  -Wl,--import-memory,--export-memory,--max-memory=67108864 example.c other.c \
  -L/root/wasm-openmp-examples/build/runtime/src -lomp \
  -lwasi-emulated-getpid \
  -o example.wasm
/root/.wasmtime/bin/wasmtime -W threads -S threads example.wasm
Hello World... from thread = 0
Hello World... from thread = 3
Hello World... from thread = 2
Hello World... from thread = 1
Hello World... from thread = 4
Hello World... from thread = 6
Hello World... from thread = 5
Hello World... from thread = 7
example.c critical_unamed: 8
example.c critical_named: 8
example.c reduction: 8
other.c critical_unamed: 8
other.c critical_named: 8
other.c reduction: 8

The open question i have is why is wasm target different here? Native platforms seem to work fine with CommonLinkage so i am curious if there is a related issue elsewhere in clang that is not emitting the proper definitions for these variables to support CommonLinkage when compiling for wasm

[llvmbot]

@llvm/pr-subscribers-flang-openmp

Author: arsnyder16 (arsnyder16)

Changes

• Separate wasi and emscripten as they have different constraints and abilities
  • Emscripten mimics Linux/POSIX by statically linking the musl runtime. This allow nearly all KMP_OS_LINUX code paths to work correctly. There are only a few places that need to be adjusted related to dynamic linking (dl_open)
• Internally link openmp globals
  • With CommonLinkage it is needed to emit them in an assembly file, now they are defined and used within each compilation unit
  • With ExternalLinkage they suffer from duplicate symbols during linking for unnamed globals like reduction/critical
  • Interestingly this aligns with the TODO comment above this code

---

Full diff: https://github.com/llvm/llvm-project/pull/95169.diff

4 Files Affected:

• (modified) llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp (+1-1)
• (modified) openmp/runtime/src/kmp_os.h (+2-2)
• (modified) openmp/runtime/src/kmp_platform.h (+7-1)
• (modified) openmp/runtime/src/z_Linux_asm.S (-20)

diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 5154c33502526..01c8ae3a6e867 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -5356,7 +5356,7 @@ OpenMPIRBuilder::getOrCreateInternalVariable(Type *Ty, const StringRef &Name,
     // create different versions of the function for different OMP internal
     // variables.
     auto Linkage = this->M.getTargetTriple().rfind("wasm32") == 0
-                       ? GlobalValue::ExternalLinkage
+                       ? GlobalValue::InternalLinkage
                        : GlobalValue::CommonLinkage;
     auto *GV = new GlobalVariable(M, Ty, /*IsConstant=*/false, Linkage,
                                   Constant::getNullValue(Ty), Elem.first(),
diff --git a/openmp/runtime/src/kmp_os.h b/openmp/runtime/src/kmp_os.h
index 24b40ed32d988..2252f5e7e97a7 100644
--- a/openmp/runtime/src/kmp_os.h
+++ b/openmp/runtime/src/kmp_os.h
@@ -77,7 +77,7 @@
 
 #if (KMP_OS_LINUX || KMP_OS_WINDOWS || KMP_OS_FREEBSD || KMP_OS_NETBSD ||      \
      KMP_OS_DRAGONFLY || KMP_OS_AIX) &&                                        \
-    !KMP_OS_WASI
+    !KMP_OS_WASI && !KMP_OS_EMSCRIPTEN
 #define KMP_AFFINITY_SUPPORTED 1
 #if KMP_OS_WINDOWS && KMP_ARCH_X86_64
 #define KMP_GROUP_AFFINITY 1
@@ -1293,7 +1293,7 @@ bool __kmp_atomic_compare_store_rel(std::atomic<T> *p, T expected, T desired) {
 extern void *__kmp_lookup_symbol(const char *name, bool next = false);
 #define KMP_DLSYM(name) __kmp_lookup_symbol(name)
 #define KMP_DLSYM_NEXT(name) __kmp_lookup_symbol(name, true)
-#elif KMP_OS_WASI
+#elif KMP_OS_WASI || KMP_OS_EMSCRIPTEN
 #define KMP_DLSYM(name) nullptr
 #define KMP_DLSYM_NEXT(name) nullptr
 #else
diff --git a/openmp/runtime/src/kmp_platform.h b/openmp/runtime/src/kmp_platform.h
index 200fdf697dd0e..9c2215140467d 100644
--- a/openmp/runtime/src/kmp_platform.h
+++ b/openmp/runtime/src/kmp_platform.h
@@ -25,6 +25,7 @@
 #define KMP_OS_HURD 0
 #define KMP_OS_SOLARIS 0
 #define KMP_OS_WASI 0
+#define KMP_OS_EMSCRIPTEN 0
 #define KMP_OS_UNIX 0 /* disjunction of KMP_OS_LINUX, KMP_OS_DARWIN etc. */
 
 #ifdef _WIN32
@@ -44,6 +45,11 @@
 #elif (defined __linux__)
 #undef KMP_OS_LINUX
 #define KMP_OS_LINUX 1
+#elif defined(__EMSCRIPTEN__)
+#undef KMP_OS_LINUX
+#undef KMP_OS_EMSCRIPTEN
+#define KMP_OS_LINUX 1
+#define KMP_OS_EMSCRIPTEN 1
 #else
 #endif
 
@@ -77,7 +83,7 @@
 #define KMP_OS_SOLARIS 1
 #endif
 
-#if (defined __wasi__) || (defined __EMSCRIPTEN__)
+#if (defined __wasi__)
 #undef KMP_OS_WASI
 #define KMP_OS_WASI 1
 #endif
diff --git a/openmp/runtime/src/z_Linux_asm.S b/openmp/runtime/src/z_Linux_asm.S
index 5b614e26a8337..f119f64647daa 100644
--- a/openmp/runtime/src/z_Linux_asm.S
+++ b/openmp/runtime/src/z_Linux_asm.S
@@ -2452,23 +2452,3 @@ KMP_PREFIX_UNDERSCORE(__kmp_unnamed_critical_addr):
 .section .note.GNU-stack,"",@progbits
 # endif
 #endif
-
-#if KMP_ARCH_WASM
-.data
-.global .gomp_critical_user_
-.global .gomp_critical_user_.var
-.global .gomp_critical_user_.reduction.var
-.global __kmp_unnamed_critical_addr
-.gomp_critical_user_:
-.zero 4
-.size .gomp_critical_user_, 4
-.gomp_critical_user_.var:
-.zero 4
-.size .gomp_critical_user_.var, 4
-.gomp_critical_user_.reduction.var:
-.zero 4
-.size .gomp_critical_user_.reduction.var, 4
-__kmp_unnamed_critical_addr:
-    .4byte .gomp_critical_user_
-    .size __kmp_unnamed_critical_addr, 4
-#endif

[abrown]

cc: @jpeyton52, can you take a look at this?

[arsnyder16]

@abrown @jpeyton52 Any updates here?

[jpeyton52]

The changes look fine to me.

[arsnyder16]

The changes look fine to me.

Thanks, good to merge?

[arsnyder16]

@jpeyton52 Can we get this merged?

[jpeyton52]

Sorry about that! It has been merged.

[arsnyder16]

@sbc100 @kripken What is the process required for this change(clang update) to land in the next emsdk version?

[kripken]

@arsnyder16 Once something lands in LLVM, it is automatically picked up by Emscripten's tip of tree builds (which you can install with emsdk install tot, though it can take a few hours for tests and builds to complete). It will also be included automatically in the next official emsdk release.