[clang-linker-wrapper] Add ELF packaging for spirv64-intel OpenMP images #125737

sarnex · 2025-02-04T18:17:16Z

Add manual ELF packaging for spirv64-intel images as there is no SPIR-V linker available. This format will be expected by the runtime plugin we will submit in the future and is compatible with the format we already use downstream.

github-actions · 2025-02-04T18:28:16Z

✅ With the latest revision this PR passed the C/C++ code formatter.

Signed-off-by: Sarnie, Nick <[email protected]>

llvmbot · 2025-02-05T15:23:29Z

@llvm/pr-subscribers-clang

Author: Nick Sarnie (sarnex)

Changes

Add manual ELF packaging for spirv64-intel images. This format will be expected by the runtime plugin we will submit in the future.

Full diff: https://github.com/llvm/llvm-project/pull/125737.diff

8 Files Affected:

(added) clang/test/Tooling/Inputs/clang-linker-wrapper-spirv-elf.o ()
(added) clang/test/Tooling/clang-linker-wrapper-spirv-elf.cpp (+14)
(modified) clang/test/Tooling/lit.local.cfg (+6)
(modified) clang/test/lit.site.cfg.py.in (+1)
(modified) clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp (+15)
(modified) llvm/include/llvm/Frontend/Offloading/Utility.h (+6)
(modified) llvm/lib/Frontend/Offloading/CMakeLists.txt (+1)
(modified) llvm/lib/Frontend/Offloading/Utility.cpp (+85)

diff --git a/clang/test/Tooling/Inputs/clang-linker-wrapper-spirv-elf.o b/clang/test/Tooling/Inputs/clang-linker-wrapper-spirv-elf.o
new file mode 100644
index 000000000000000..3e5bddcedfff163
Binary files /dev/null and b/clang/test/Tooling/Inputs/clang-linker-wrapper-spirv-elf.o differ
diff --git a/clang/test/Tooling/clang-linker-wrapper-spirv-elf.cpp b/clang/test/Tooling/clang-linker-wrapper-spirv-elf.cpp
new file mode 100644
index 000000000000000..602132172d0248f
--- /dev/null
+++ b/clang/test/Tooling/clang-linker-wrapper-spirv-elf.cpp
@@ -0,0 +1,14 @@
+// Verify the ELF packaging of OpenMP SPIR-V device images.
+// REQUIRES: system-linux
+// REQUIRES: spirv-tools
+// RUN: mkdir -p %t_tmp
+// RUN: cd %t_tmp
+// RUN: not clang-linker-wrapper -o a.out %S/Inputs/clang-linker-wrapper-spirv-elf.o --save-temps --linker-path=ld
+// RUN: clang-offload-packager --image=triple=spirv64-intel,kind=openmp,file=%t.elf  %t_tmp/a.out.openmp.image.wrapper.o
+// RUN: llvm-readelf -t %t.elf | FileCheck -check-prefix=CHECK-SECTION %s
+// RUN: llvm-readelf -n %t.elf | FileCheck -check-prefix=CHECK-NOTES %s
+
+// CHECK-SECTION: .note.inteloneompoffload
+// CHECK-SECTION: __openmp_offload_spirv_0
+
+// CHECK-NOTES-COUNT-3: INTELONEOMPOFFLOAD
diff --git a/clang/test/Tooling/lit.local.cfg b/clang/test/Tooling/lit.local.cfg
index 4cd8ba72fa76715..bc2a096c8f64f88 100644
--- a/clang/test/Tooling/lit.local.cfg
+++ b/clang/test/Tooling/lit.local.cfg
@@ -1,2 +1,8 @@
 if not config.root.clang_staticanalyzer:
     config.unsupported = True
+
+if config.spirv_tools_tests:
+    config.available_features.add("spirv-tools")
+    config.substitutions.append(("spirv-dis", os.path.join(config.llvm_tools_dir, "spirv-dis")))
+    config.substitutions.append(("spirv-val", os.path.join(config.llvm_tools_dir, "spirv-val")))
+    config.substitutions.append(("spirv-as", os.path.join(config.llvm_tools_dir, "spirv-as")))
diff --git a/clang/test/lit.site.cfg.py.in b/clang/test/lit.site.cfg.py.in
index ae8b927624e23f8..ce10e9128a1dfe1 100644
--- a/clang/test/lit.site.cfg.py.in
+++ b/clang/test/lit.site.cfg.py.in
@@ -43,6 +43,7 @@ config.llvm_external_lit = path(r"@LLVM_EXTERNAL_LIT@")
 config.standalone_build = @CLANG_BUILT_STANDALONE@
 config.ppc_linux_default_ieeelongdouble = @PPC_LINUX_DEFAULT_IEEELONGDOUBLE@
 config.have_llvm_driver = @LLVM_TOOL_LLVM_DRIVER_BUILD@
+config.spirv_tools_tests = "@LLVM_INCLUDE_SPIRV_TOOLS_TESTS@"
 config.substitutions.append(("%llvm-version-major", "@LLVM_VERSION_MAJOR@"))
 
 import lit.llvm
diff --git a/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp b/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp
index c92590581a645cc..d7d6418a4b5e2e0 100644
--- a/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp
+++ b/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp
@@ -605,6 +605,17 @@ Expected<StringRef> linkDevice(ArrayRef<StringRef> InputFiles,
   }
 }
 
+Error containerizeRawImage(std::unique_ptr<MemoryBuffer> &Img, OffloadKind Kind,
+                           const ArgList &Args) {
+  llvm::Triple Triple(Args.getLastArgValue(OPT_triple_EQ));
+  if (Kind != OFK_OpenMP || !Triple.isSPIRV() ||
+      Triple.getVendor() != llvm::Triple::Intel)
+    return Error::success();
+  if (Error E = offloading::intel::containerizeOpenMPSPIRVImage(Img))
+    return E;
+  return Error::success();
+}
+
 Expected<StringRef> writeOffloadFile(const OffloadFile &File) {
   const OffloadBinary &Binary = *File.getBinary();
 
@@ -960,6 +971,10 @@ Expected<SmallVector<StringRef>> linkAndWrapDeviceFiles(
           return createFileError(*OutputOrErr, EC);
       }
 
+      // Manually containerize offloading images not in ELF format.
+      if (Error E = containerizeRawImage(*FileOrErr, Kind, LinkerArgs))
+        return E;
+
       std::scoped_lock<decltype(ImageMtx)> Guard(ImageMtx);
       OffloadingImage TheImage{};
       TheImage.TheImageKind =
diff --git a/llvm/include/llvm/Frontend/Offloading/Utility.h b/llvm/include/llvm/Frontend/Offloading/Utility.h
index 7932fd5acbe1e26..9140371a8c2ed21 100644
--- a/llvm/include/llvm/Frontend/Offloading/Utility.h
+++ b/llvm/include/llvm/Frontend/Offloading/Utility.h
@@ -10,6 +10,7 @@
 #define LLVM_FRONTEND_OFFLOADING_UTILITY_H
 
 #include <cstdint>
+#include <memory>
 
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
@@ -152,6 +153,11 @@ Error getAMDGPUMetaDataFromImage(MemoryBufferRef MemBuffer,
                                  StringMap<AMDGPUKernelMetaData> &KernelInfoMap,
                                  uint16_t &ELFABIVersion);
 } // namespace amdgpu
+namespace intel {
+/// Containerizes an offloading binary into the ELF binary format expected by
+/// the Intel runtime offload plugin.
+Error containerizeOpenMPSPIRVImage(std::unique_ptr<MemoryBuffer> &Binary);
+} // namespace intel
 } // namespace offloading
 } // namespace llvm
 
diff --git a/llvm/lib/Frontend/Offloading/CMakeLists.txt b/llvm/lib/Frontend/Offloading/CMakeLists.txt
index ce445ad9cc4cb60..8e1ede9c72b391a 100644
--- a/llvm/lib/Frontend/Offloading/CMakeLists.txt
+++ b/llvm/lib/Frontend/Offloading/CMakeLists.txt
@@ -12,6 +12,7 @@ add_llvm_component_library(LLVMFrontendOffloading
   Core
   BinaryFormat
   Object
+  ObjectYAML
   Support
   TransformUtils
   TargetParser
diff --git a/llvm/lib/Frontend/Offloading/Utility.cpp b/llvm/lib/Frontend/Offloading/Utility.cpp
index 8117a42b8a45cd1..f9c74ab975d1023 100644
--- a/llvm/lib/Frontend/Offloading/Utility.cpp
+++ b/llvm/lib/Frontend/Offloading/Utility.cpp
@@ -15,6 +15,8 @@
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Object/ELFObjectFile.h"
+#include "llvm/ObjectYAML/ELFYAML.h"
+#include "llvm/ObjectYAML/yaml2obj.h"
 #include "llvm/Support/MemoryBufferRef.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 
@@ -373,3 +375,86 @@ Error llvm::offloading::amdgpu::getAMDGPUMetaDataFromImage(
   }
   return Error::success();
 }
+Error offloading::intel::containerizeOpenMPSPIRVImage(
+    std::unique_ptr<MemoryBuffer> &Img) {
+  constexpr char INTEL_ONEOMP_OFFLOAD_VERSION[] = "1.0";
+  constexpr int NT_INTEL_ONEOMP_OFFLOAD_VERSION = 1;
+  constexpr int NT_INTEL_ONEOMP_OFFLOAD_IMAGE_COUNT = 2;
+  constexpr int NT_INTEL_ONEOMP_OFFLOAD_IMAGE_AUX = 3;
+
+  // Start creating notes for the ELF container.
+  std::vector<ELFYAML::NoteEntry> Notes;
+  std::string Version = toHex(INTEL_ONEOMP_OFFLOAD_VERSION);
+  Notes.emplace_back(ELFYAML::NoteEntry{"INTELONEOMPOFFLOAD",
+                                        yaml::BinaryRef(Version),
+                                        NT_INTEL_ONEOMP_OFFLOAD_VERSION});
+
+  // The AuxInfo string will hold auxiliary information for the image.
+  // ELFYAML::NoteEntry structures will hold references to the
+  // string, so we have to make sure the string is valid.
+  std::string AuxInfo;
+
+  // TODO: Pass compile/link opts
+  StringRef CompileOpts = "";
+  StringRef LinkOpts = "";
+
+  unsigned ImageFmt = 1; // SPIR-V format
+
+  AuxInfo = toHex((Twine(0) + Twine('\0') + Twine(ImageFmt) + Twine('\0') +
+                   CompileOpts + Twine('\0') + LinkOpts)
+                      .str());
+  Notes.emplace_back(ELFYAML::NoteEntry{"INTELONEOMPOFFLOAD",
+                                        yaml::BinaryRef(AuxInfo),
+                                        NT_INTEL_ONEOMP_OFFLOAD_IMAGE_AUX});
+
+  std::string ImgCount = toHex(Twine(1).str()); // always one image per ELF
+  Notes.emplace_back(ELFYAML::NoteEntry{"INTELONEOMPOFFLOAD",
+                                        yaml::BinaryRef(ImgCount),
+                                        NT_INTEL_ONEOMP_OFFLOAD_IMAGE_COUNT});
+
+  std::string YamlFile;
+  llvm::raw_string_ostream YamlFileStream(YamlFile);
+
+  // Write YAML template file.
+  {
+    // We use 64-bit little-endian ELF currently.
+    ELFYAML::FileHeader Header{};
+    Header.Class = ELF::ELFCLASS64;
+    Header.Data = ELF::ELFDATA2LSB;
+    Header.Type = ELF::ET_DYN;
+    // Use an existing Intel machine type as there is not one specifically for
+    // Intel GPUs.
+    Header.Machine = ELF::EM_IA_64;
+
+    // Create a section with notes.
+    ELFYAML::NoteSection Section{};
+    Section.Type = ELF::SHT_NOTE;
+    Section.AddressAlign = 0;
+    Section.Name = ".note.inteloneompoffload";
+    Section.Notes.emplace(std::move(Notes));
+
+    ELFYAML::Object Object{};
+    Object.Header = Header;
+    Object.Chunks.push_back(
+        std::make_unique<ELFYAML::NoteSection>(std::move(Section)));
+
+    // Create the section that will hold the image
+    ELFYAML::RawContentSection ImageSection{};
+    ImageSection.Type = ELF::SHT_PROGBITS;
+    ImageSection.AddressAlign = 0;
+    std::string Name = "__openmp_offload_spirv_0";
+    ImageSection.Name = Name;
+    ImageSection.Content =
+        llvm::yaml::BinaryRef(arrayRefFromStringRef(Img->getBuffer()));
+    Object.Chunks.push_back(
+        std::make_unique<ELFYAML::RawContentSection>(std::move(ImageSection)));
+    Error Err = Error::success();
+    llvm::yaml::yaml2elf(
+        Object, YamlFileStream,
+        [&Err](const Twine &Msg) { Err = createStringError(Msg); }, UINT64_MAX);
+    if (Err)
+      return Err;
+  }
+  Img = MemoryBuffer::getMemBufferCopy(YamlFile);
+  return Error::success();
+}

clang/test/Tooling/Inputs/clang-linker-wrapper-spirv-elf.o

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp

llvm/lib/Frontend/Offloading/Utility.cpp

llvm/include/llvm/Frontend/Offloading/Utility.h

Signed-off-by: Sarnie, Nick <[email protected]>

dyung · 2025-02-06T16:54:50Z

Hi @sarnex, the test you added seems to be failing on a buildbot, can you take a look?

https://lab.llvm.org/buildbot/#/builders/144/builds/17530

******************** TEST 'Clang :: Tooling/clang-linker-wrapper-spirv-elf.cpp' FAILED ********************
Exit Code: 1
Command Output (stderr):
--
RUN: at line 4: mkdir -p /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Tooling/Output/clang-linker-wrapper-spirv-elf.cpp.tmp_tmp
+ mkdir -p /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Tooling/Output/clang-linker-wrapper-spirv-elf.cpp.tmp_tmp
RUN: at line 5: cd /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Tooling/Output/clang-linker-wrapper-spirv-elf.cpp.tmp_tmp
+ cd /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Tooling/Output/clang-linker-wrapper-spirv-elf.cpp.tmp_tmp
RUN: at line 6: /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/clang --driver-mode=g++ -fopenmp -fopenmp-targets=spirv64-intel -nogpulib -c -o /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Tooling/Output/clang-linker-wrapper-spirv-elf.cpp.tmp_clang-linker-wrapper-spirv-elf.o /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Tooling/clang-linker-wrapper-spirv-elf.cpp
+ /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/clang --driver-mode=g++ -fopenmp -fopenmp-targets=spirv64-intel -nogpulib -c -o /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Tooling/Output/clang-linker-wrapper-spirv-elf.cpp.tmp_clang-linker-wrapper-spirv-elf.o /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Tooling/clang-linker-wrapper-spirv-elf.cpp
clang: error: unable to execute command: Executable "llvm-spirv" doesn't exist!
clang: error: llvm-spirv command failed with exit code 1 (use -v to see invocation)
--
********************

sarnex · 2025-02-06T16:57:17Z

looking

DavidSpickett · 2025-02-06T17:11:50Z

I wonder if it is only happening on a rebuild not a clean build. This one was clean:
https://lab.llvm.org/buildbot/#/builders/65/builds/11869

But your test did not fail (I assume did not run). This one is a rebuild:
https://lab.llvm.org/buildbot/#/builders/65/builds/11870

Where it does fail. I have seen this before and it was because I had missed some part of adding a new feature to lit.You may want to look at other variables like LLVM_INCLUDE_SPIRV_TOOLS_TESTS and make sure you have added all the same references.

sarnex · 2025-02-06T17:17:46Z

yeah my test should only run if LLVM_INCLUDE_SPIRV_TOOLS_TESTS is set, which does not appear to be the case, so i'm surprised it's running. it indeed did not run in the clean build.

UNSUPPORTED: Clang :: Tooling/clang-linker-wrapper-spirv-elf.cpp (20383 of 84127)

the lit feature i added should only be enabled if LLVM_INCLUDE_SPIRV_TOOLS_TESTS is set, so to be honest i have no idea what the problem is

jhuber6 · 2025-02-06T17:18:45Z

yeah my test should only run if LLVM_INCLUDE_SPIRV_TOOLS_TESTS is set, which does not appear to be the case, so i'm surprised it's running. it indeed did not run in the clean build.
UNSUPPORTED: Clang :: Tooling/clang-linker-wrapper-spirv-elf.cpp (20383 of 84127)
the lit feature i added should only be enabled if LLVM_INCLUDE_SPIRV_TOOLS_TESTS is set, so to be honest i have no idea what the problem is

Might just be a stale config, I don't know what causes those rules to be regenerated.

sarnex · 2025-02-06T17:19:13Z

what should i do in that case? can i just ignore the failure or is there a way to clear the build cache?

jhuber6 · 2025-02-06T17:20:58Z

what should i do in that case? can i just ignore the failure or is there a way to clear the build cache?

I got the failure locally as well, unsure, maybe there's a dependency missing somewhere.

sarnex · 2025-02-06T17:22:04Z

ok cool if you can repro it it's almost surely a real issue. can you share the repro steps? my local repro has it as unsupported as i expect

dyung · 2025-02-06T17:22:43Z

If you cannot fix this soon, can you revert to get the bots back to green while you investigate?

sarnex · 2025-02-06T17:23:12Z

yes i will revert if i dont have a fix pr within 30 minutes

DavidSpickett · 2025-02-06T17:27:12Z

I have seen this before

a2bd5db

sarnex · 2025-02-06T17:31:13Z

thanks a lot, i think i see the problem, but would help if i could repro :P

sarnex · 2025-02-06T17:42:57Z

#126098

Fix for lit fail from #125737 Signed-off-by: Sarnie, Nick <[email protected]>

…26098) Fix for lit fail from llvm/llvm-project#125737 Signed-off-by: Sarnie, Nick <[email protected]>

…ges (llvm#125737) Add manual ELF packaging for `spirv64-intel` images as there is no SPIR-V linker available. This format will be expected by the runtime plugin we will submit in the future and is compatible with the format we already use downstream. --------- Signed-off-by: Sarnie, Nick <[email protected]>

Fix for lit fail from llvm#125737 Signed-off-by: Sarnie, Nick <[email protected]>

[clang-linker-wrapper] Add ELF packaging for spirv64-intel OpenMP images

d4b3358

Signed-off-by: Sarnie, Nick <[email protected]>

sarnex force-pushed the elf branch from b58c680 to d4b3358 Compare February 4, 2025 21:49

sarnex marked this pull request as ready for review February 5, 2025 15:22

llvmbot added the clang Clang issues not falling into any other category label Feb 5, 2025

sarnex mentioned this pull request Feb 5, 2025

[offload] Extend LIT infrastructure to allow using more tools #124636

Closed

sarnex requested review from shiltian and jhuber6 February 5, 2025 15:23

jhuber6 reviewed Feb 5, 2025

View reviewed changes

clang/test/Tooling/Inputs/clang-linker-wrapper-spirv-elf.o Outdated Show resolved Hide resolved

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp Outdated Show resolved Hide resolved

llvm/lib/Frontend/Offloading/Utility.cpp Outdated Show resolved Hide resolved

jhuber6 reviewed Feb 5, 2025

View reviewed changes

llvm/include/llvm/Frontend/Offloading/Utility.h Show resolved Hide resolved

improve lit test, apply code-style feedback

442ed9f

Signed-off-by: Sarnie, Nick <[email protected]>

jhuber6 approved these changes Feb 5, 2025

View reviewed changes

sarnex merged commit f7b3559 into llvm:main Feb 6, 2025
8 checks passed

sarnex mentioned this pull request Feb 6, 2025

[clang-linker-wrapper][lit] Fix SPIR-V OpenMP ELF test #126098

Merged

jhuber6 pushed a commit that referenced this pull request Feb 6, 2025

[clang-linker-wrapper][lit] Fix SPIR-V OpenMP ELF test (#126098)

4717bab

Fix for lit fail from #125737 Signed-off-by: Sarnie, Nick <[email protected]>

github-actions bot pushed a commit to arm/arm-toolchain that referenced this pull request Feb 6, 2025

Automerge: [clang-linker-wrapper][lit] Fix SPIR-V OpenMP ELF test (#1…

fe9cf3a

…26098) Fix for lit fail from llvm/llvm-project#125737 Signed-off-by: Sarnie, Nick <[email protected]>

Icohedron pushed a commit to Icohedron/llvm-project that referenced this pull request Feb 11, 2025

[clang-linker-wrapper][lit] Fix SPIR-V OpenMP ELF test (llvm#126098)

f5038da

Fix for lit fail from llvm#125737 Signed-off-by: Sarnie, Nick <[email protected]>

[clang-linker-wrapper] Add ELF packaging for spirv64-intel OpenMP images #125737

[clang-linker-wrapper] Add ELF packaging for spirv64-intel OpenMP images #125737

Uh oh!

Conversation

sarnex commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Feb 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dyung commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025

Uh oh!

DavidSpickett commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025

Uh oh!

jhuber6 commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025

Uh oh!

jhuber6 commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dyung commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidSpickett commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025

Uh oh!

sarnex commented Feb 6, 2025

Uh oh!

Uh oh!

sarnex commented Feb 4, 2025 •

edited

Loading

github-actions bot commented Feb 4, 2025 •

edited

Loading

sarnex commented Feb 6, 2025 •

edited

Loading

sarnex commented Feb 6, 2025 •

edited

Loading