Extend llvm objdump fatbin #140286
Merged
Extend llvm objdump fatbin #140286
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With the intention to provide a common API for offloading, this extension to the existing LLVM Offloading API adds support for Binary Fatbin Bundles; moving some support from the Clang offloading API. The intention is to add functionality to LLVM tooling for Binary Fatbin Bundles in subsequent commits. Change-Id: I907fdcbcd0545162a0ce1cf17ebf7c9f3a4dbde6
|
@llvm/pr-subscribers-llvm-binary-utilities Author: David Salinas (david-salinas) ChangesPatch is 69.01 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/140286.diff 10 Files Affected:
diff --git a/llvm/docs/CommandGuide/llvm-objdump.rst b/llvm/docs/CommandGuide/llvm-objdump.rst
index ab9f583e96ec6..5e5eaccecd2b7 100644
--- a/llvm/docs/CommandGuide/llvm-objdump.rst
+++ b/llvm/docs/CommandGuide/llvm-objdump.rst
@@ -217,7 +217,7 @@ OPTIONS
.. option:: --offloading
- Display the content of the LLVM offloading section.
+ Display the content of the LLVM offloading sections and HIP offload bundles.
.. option:: --prefix=<prefix>
diff --git a/llvm/include/llvm/Object/OffloadBundle.h b/llvm/include/llvm/Object/OffloadBundle.h
new file mode 100644
index 0000000000000..b6be7c3296cc1
--- /dev/null
+++ b/llvm/include/llvm/Object/OffloadBundle.h
@@ -0,0 +1,212 @@
+//===- OffloadBundle.h - Utilities for offload bundles---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===-------------------------------------------------------------------------===//
+//
+// This file contains the binary format used for budingling device metadata with
+// an associated device image. The data can then be stored inside a host object
+// file to create a fat binary and read by the linker. This is intended to be a
+// thin wrapper around the image itself. If this format becomes sufficiently
+// complex it should be moved to a standard binary format like msgpack or ELF.
+//
+//===-------------------------------------------------------------------------===//
+
+#ifndef LLVM_OBJECT_OFFLOADBUNDLE_H
+#define LLVM_OBJECT_OFFLOADBUNDLE_H
+
+#include "llvm/ADT/MapVector.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Object/Binary.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/Support/Compression.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include <memory>
+
+namespace llvm {
+
+namespace object {
+
+class CompressedOffloadBundle {
+private:
+ static inline const size_t MagicSize = 4;
+ static inline const size_t VersionFieldSize = sizeof(uint16_t);
+ static inline const size_t MethodFieldSize = sizeof(uint16_t);
+ static inline const size_t FileSizeFieldSize = sizeof(uint32_t);
+ static inline const size_t UncompressedSizeFieldSize = sizeof(uint32_t);
+ static inline const size_t HashFieldSize = sizeof(uint64_t);
+ static inline const size_t V1HeaderSize =
+ MagicSize + VersionFieldSize + MethodFieldSize +
+ UncompressedSizeFieldSize + HashFieldSize;
+ static inline const size_t V2HeaderSize =
+ MagicSize + VersionFieldSize + FileSizeFieldSize + MethodFieldSize +
+ UncompressedSizeFieldSize + HashFieldSize;
+ static inline const llvm::StringRef MagicNumber = "CCOB";
+ static inline const uint16_t Version = 2;
+
+public:
+ static llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
+ compress(llvm::compression::Params P, const llvm::MemoryBuffer &Input,
+ bool Verbose = false);
+ static llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
+ decompress(llvm::MemoryBufferRef &Input, bool Verbose = false);
+};
+
+/// Bundle entry in binary clang-offload-bundler format.
+struct OffloadBundleEntry {
+ uint64_t Offset = 0u;
+ uint64_t Size = 0u;
+ uint64_t IDLength = 0u;
+ StringRef ID;
+ OffloadBundleEntry(uint64_t O, uint64_t S, uint64_t I, StringRef T)
+ : Offset(O), Size(S), IDLength(I), ID(T) {}
+ void dumpInfo(raw_ostream &OS) {
+ OS << "Offset = " << Offset << ", Size = " << Size
+ << ", ID Length = " << IDLength << ", ID = " << ID;
+ }
+ void dumpURI(raw_ostream &OS, StringRef FilePath) {
+ OS << ID.data() << "\tfile://" << FilePath << "#offset=" << Offset
+ << "&size=" << Size << "\n";
+ }
+};
+
+/// Fat binary embedded in object files in clang-offload-bundler format
+class OffloadBundleFatBin {
+
+ uint64_t Size = 0u;
+ StringRef FileName;
+ uint64_t NumberOfEntries;
+ SmallVector<OffloadBundleEntry> Entries;
+
+public:
+ SmallVector<OffloadBundleEntry> getEntries() { return Entries; }
+ uint64_t getSize() const { return Size; }
+ StringRef getFileName() const { return FileName; }
+ uint64_t getNumEntries() const { return NumberOfEntries; }
+
+ static Expected<std::unique_ptr<OffloadBundleFatBin>>
+ create(MemoryBufferRef, uint64_t SectionOffset, StringRef FileName);
+ Error extractBundle(const ObjectFile &Source);
+
+ Error dumpEntryToCodeObject();
+
+ Error readEntries(StringRef Section, uint64_t SectionOffset);
+ void dumpEntries() {
+ for (OffloadBundleEntry &Entry : Entries)
+ Entry.dumpInfo(outs());
+ }
+
+ void printEntriesAsURI() {
+ for (OffloadBundleEntry &Entry : Entries)
+ Entry.dumpURI(outs(), FileName);
+ }
+
+ OffloadBundleFatBin(MemoryBufferRef Source, StringRef File)
+ : FileName(File), NumberOfEntries(0),
+ Entries(SmallVector<OffloadBundleEntry>()) {}
+
+ SmallVector<OffloadBundleEntry> entryIDContains(StringRef Str) {
+
+ SmallVector<OffloadBundleEntry> Found = SmallVector<OffloadBundleEntry>();
+ llvm::transform(Entries, std::back_inserter(Found), [Str](auto &X) {
+ if (X.ID.contains(Str))
+ return X;
+ });
+ return Found;
+ }
+};
+
+enum UriTypeT { FILE_URI, MEMORY_URI };
+
+struct OffloadBundleURI {
+ int64_t Offset = 0;
+ int64_t Size = 0;
+ uint64_t ProcessID = 0;
+ StringRef FileName;
+ UriTypeT URIType;
+
+ // Constructors
+ // TODO: add a Copy ctor ?
+ OffloadBundleURI(StringRef File, int64_t Off, int64_t Size)
+ : Offset(Off), Size(Size), ProcessID(0), FileName(File),
+ URIType(FILE_URI) {}
+
+public:
+ static Expected<std::unique_ptr<OffloadBundleURI>>
+ createOffloadBundleURI(StringRef Str, UriTypeT Type) {
+ switch (Type) {
+ case FILE_URI:
+ return createFileURI(Str);
+ break;
+ case MEMORY_URI:
+ return createMemoryURI(Str);
+ break;
+ default:
+ return createStringError(object_error::parse_failed,
+ "Unrecognized URI type");
+ }
+ }
+
+ static Expected<std::unique_ptr<OffloadBundleURI>>
+ createFileURI(StringRef Str) {
+ int64_t O = 0;
+ int64_t S = 0;
+
+ if (!Str.consume_front("file://"))
+ return createStringError(object_error::parse_failed,
+ "Reading type of URI");
+
+ StringRef FilePathname =
+ Str.take_until([](char C) { return (C == '#') || (C == '?'); });
+ Str = Str.drop_front(FilePathname.size());
+
+ if (!Str.consume_front("#offset="))
+ return createStringError(object_error::parse_failed,
+ "Reading 'offset' in URI");
+
+ StringRef OffsetStr = Str.take_until([](char C) { return C == '&'; });
+ OffsetStr.getAsInteger(10, O);
+ Str = Str.drop_front(OffsetStr.size());
+
+ if (Str.consume_front("&size="))
+ return createStringError(object_error::parse_failed,
+ "Reading 'size' in URI");
+
+ Str.getAsInteger(10, S);
+ std::unique_ptr<OffloadBundleURI> OffloadingURI(
+ new OffloadBundleURI(FilePathname, O, S));
+ // SALINAS return OffloadingURI;
+ return std::move(OffloadingURI);
+ }
+
+ static Expected<std::unique_ptr<OffloadBundleURI>>
+ createMemoryURI(StringRef Str) {
+ // TODO: add parseMemoryURI type
+ return createStringError(object_error::parse_failed,
+ "Memory Type URI is not currently supported.");
+ }
+
+ StringRef getFileName() const { return FileName; }
+};
+
+/// Extracts fat binary in binary clang-offload-bundler format from object \p
+/// Obj and return it in \p Bundles
+Error extractOffloadBundleFatBinary(
+ const ObjectFile &Obj, SmallVectorImpl<OffloadBundleFatBin> &Bundles);
+
+/// Extract code object memory from the given \p Source object file at \p Offset
+/// and of \p Size, and copy into \p OutputFileName.
+Error extractCodeObject(const ObjectFile &Source, int64_t Offset, int64_t Size,
+ StringRef OutputFileName);
+
+/// Extracts an Offload Bundle Entry given by URI
+Error extractOffloadBundleByURI(StringRef URIstr);
+
+} // namespace object
+
+} // namespace llvm
+#endif
diff --git a/llvm/lib/Object/CMakeLists.txt b/llvm/lib/Object/CMakeLists.txt
index bfb420e57a7f4..870169a83174f 100644
--- a/llvm/lib/Object/CMakeLists.txt
+++ b/llvm/lib/Object/CMakeLists.txt
@@ -22,6 +22,7 @@ add_llvm_component_library(LLVMObject
Object.cpp
ObjectFile.cpp
OffloadBinary.cpp
+ OffloadBundle.cpp
RecordStreamer.cpp
RelocationResolver.cpp
SymbolicFile.cpp
diff --git a/llvm/lib/Object/OffloadBundle.cpp b/llvm/lib/Object/OffloadBundle.cpp
new file mode 100644
index 0000000000000..5f087a5c84e8d
--- /dev/null
+++ b/llvm/lib/Object/OffloadBundle.cpp
@@ -0,0 +1,473 @@
+//===- OffloadBundle.cpp - Utilities for offload bundles---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------===//
+
+#include "llvm/Object/OffloadBundle.h"
+#include "llvm/ADT/StringSwitch.h"
+#include "llvm/BinaryFormat/Magic.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IRReader/IRReader.h"
+#include "llvm/MC/StringTableBuilder.h"
+#include "llvm/Object/Archive.h"
+#include "llvm/Object/Binary.h"
+#include "llvm/Object/COFF.h"
+#include "llvm/Object/ELFObjectFile.h"
+#include "llvm/Object/Error.h"
+#include "llvm/Object/IRObjectFile.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/Support/Alignment.h"
+#include "llvm/Support/BinaryStreamReader.h"
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/Timer.h"
+
+using namespace llvm;
+using namespace llvm::object;
+
+static llvm::TimerGroup
+ OffloadBundlerTimerGroup("Offload Bundler Timer Group",
+ "Timer group for offload bundler");
+
+// Extract an Offload bundle (usually a Offload Bundle) from a fat_bin
+// section
+Error extractOffloadBundle(MemoryBufferRef Contents, uint64_t SectionOffset,
+ StringRef FileName,
+ SmallVectorImpl<OffloadBundleFatBin> &Bundles) {
+
+ uint64_t Offset = 0;
+ int64_t NextbundleStart = 0;
+
+ // There could be multiple offloading bundles stored at this section.
+ while (NextbundleStart >= 0) {
+
+ std::unique_ptr<MemoryBuffer> Buffer =
+ MemoryBuffer::getMemBuffer(Contents.getBuffer().drop_front(Offset), "",
+ /*RequiresNullTerminator=*/false);
+
+ // Create the FatBinBindle object. This will also create the Bundle Entry
+ // list info.
+ auto FatBundleOrErr =
+ OffloadBundleFatBin::create(*Buffer, SectionOffset + Offset, FileName);
+ if (!FatBundleOrErr)
+ return FatBundleOrErr.takeError();
+
+ // Add current Bundle to list.
+ Bundles.emplace_back(std::move(**FatBundleOrErr));
+
+ // Find the next bundle by searching for the magic string
+ StringRef Str = Buffer->getBuffer();
+ NextbundleStart =
+ (int64_t)Str.find(StringRef("__CLANG_OFFLOAD_BUNDLE__"), 24);
+
+ if (NextbundleStart >= 0)
+ Offset += NextbundleStart;
+ }
+
+ return Error::success();
+}
+
+Error OffloadBundleFatBin::readEntries(StringRef Buffer,
+ uint64_t SectionOffset) {
+ uint64_t NumOfEntries = 0;
+
+ BinaryStreamReader Reader(Buffer, llvm::endianness::little);
+
+ // Read the Magic String first.
+ StringRef Magic;
+ if (auto EC = Reader.readFixedString(Magic, 24))
+ return errorCodeToError(object_error::parse_failed);
+
+ // Read the number of Code Objects (Entries) in the current Bundle.
+ if (auto EC = Reader.readInteger(NumOfEntries))
+ return errorCodeToError(object_error::parse_failed);
+
+ NumberOfEntries = NumOfEntries;
+
+ // For each Bundle Entry (code object)
+ for (uint64_t I = 0; I < NumOfEntries; I++) {
+ uint64_t EntrySize;
+ uint64_t EntryOffset;
+ uint64_t EntryIDSize;
+ StringRef EntryID;
+
+ if (auto EC = Reader.readInteger(EntryOffset))
+ return errorCodeToError(object_error::parse_failed);
+
+ if (auto EC = Reader.readInteger(EntrySize))
+ return errorCodeToError(object_error::parse_failed);
+
+ if (auto EC = Reader.readInteger(EntryIDSize))
+ return errorCodeToError(object_error::parse_failed);
+
+ if (auto EC = Reader.readFixedString(EntryID, EntryIDSize))
+ return errorCodeToError(object_error::parse_failed);
+
+ auto Entry = std::make_unique<OffloadBundleEntry>(
+ EntryOffset + SectionOffset, EntrySize, EntryIDSize, EntryID);
+
+ Entries.push_back(*Entry);
+ }
+
+ return Error::success();
+}
+
+Expected<std::unique_ptr<OffloadBundleFatBin>>
+OffloadBundleFatBin::create(MemoryBufferRef Buf, uint64_t SectionOffset,
+ StringRef FileName) {
+ if (Buf.getBufferSize() < 24)
+ return errorCodeToError(object_error::parse_failed);
+
+ // Check for magic bytes.
+ if (identify_magic(Buf.getBuffer()) != file_magic::offload_bundle)
+ return errorCodeToError(object_error::parse_failed);
+
+ OffloadBundleFatBin *TheBundle = new OffloadBundleFatBin(Buf, FileName);
+
+ // Read the Bundle Entries
+ Error Err = TheBundle->readEntries(Buf.getBuffer(), SectionOffset);
+ if (Err)
+ return errorCodeToError(object_error::parse_failed);
+
+ return std::unique_ptr<OffloadBundleFatBin>(TheBundle);
+}
+
+Error OffloadBundleFatBin::extractBundle(const ObjectFile &Source) {
+ // This will extract all entries in the Bundle
+ for (OffloadBundleEntry &Entry : Entries) {
+
+ if (Entry.Size == 0)
+ continue;
+
+ // create output file name. Which should be
+ // <fileName>-offset<Offset>-size<Size>.co"
+ std::string Str = getFileName().str() + "-offset" + itostr(Entry.Offset) +
+ "-size" + itostr(Entry.Size) + ".co";
+ if (Error Err = object::extractCodeObject(Source, Entry.Offset, Entry.Size,
+ StringRef(Str)))
+ return Err;
+ }
+
+ return Error::success();
+}
+
+Error object::extractOffloadBundleFatBinary(
+ const ObjectFile &Obj, SmallVectorImpl<OffloadBundleFatBin> &Bundles) {
+ assert((Obj.isELF() || Obj.isCOFF()) && "Invalid file type");
+
+ // Iterate through Sections until we find an offload_bundle section.
+ for (SectionRef Sec : Obj.sections()) {
+ Expected<StringRef> Buffer = Sec.getContents();
+ if (!Buffer)
+ return Buffer.takeError();
+
+ // If it does not start with the reserved suffix, just skip this section.
+ if ((llvm::identify_magic(*Buffer) == llvm::file_magic::offload_bundle) ||
+ (llvm::identify_magic(*Buffer) ==
+ llvm::file_magic::offload_bundle_compressed)) {
+
+ uint64_t SectionOffset = 0;
+ if (Obj.isELF()) {
+ SectionOffset = ELFSectionRef(Sec).getOffset();
+ } else if (Obj.isCOFF()) // TODO: add COFF Support
+ return createStringError(object_error::parse_failed,
+ "COFF object files not supported.\n");
+
+ MemoryBufferRef Contents(*Buffer, Obj.getFileName());
+
+ if (llvm::identify_magic(*Buffer) ==
+ llvm::file_magic::offload_bundle_compressed) {
+ // Decompress the input if necessary.
+ Expected<std::unique_ptr<MemoryBuffer>> DecompressedBufferOrErr =
+ CompressedOffloadBundle::decompress(Contents, false);
+
+ if (!DecompressedBufferOrErr)
+ return createStringError(
+ inconvertibleErrorCode(),
+ "Failed to decompress input: " +
+ llvm::toString(DecompressedBufferOrErr.takeError()));
+
+ MemoryBuffer &DecompressedInput = **DecompressedBufferOrErr;
+ if (Error Err = extractOffloadBundle(DecompressedInput, SectionOffset,
+ Obj.getFileName(), Bundles))
+ return Err;
+ } else {
+ if (Error Err = extractOffloadBundle(Contents, SectionOffset,
+ Obj.getFileName(), Bundles))
+ return Err;
+ }
+ }
+ }
+ return Error::success();
+}
+
+Error object::extractCodeObject(const ObjectFile &Source, int64_t Offset,
+ int64_t Size, StringRef OutputFileName) {
+ Expected<std::unique_ptr<FileOutputBuffer>> BufferOrErr =
+ FileOutputBuffer::create(OutputFileName, Size);
+
+ if (!BufferOrErr)
+ return BufferOrErr.takeError();
+
+ Expected<MemoryBufferRef> InputBuffOrErr = Source.getMemoryBufferRef();
+ if (Error Err = InputBuffOrErr.takeError())
+ return Err;
+
+ std::unique_ptr<FileOutputBuffer> Buf = std::move(*BufferOrErr);
+ std::copy(InputBuffOrErr->getBufferStart() + Offset,
+ InputBuffOrErr->getBufferStart() + Offset + Size,
+ Buf->getBufferStart());
+ if (Error E = Buf->commit())
+ return E;
+
+ return Error::success();
+}
+
+// given a file name, offset, and size, extract data into a code object file,
+// into file <SourceFile>-offset<Offset>-size<Size>.co
+Error object::extractOffloadBundleByURI(StringRef URIstr) {
+ // create a URI object
+ Expected<std::unique_ptr<OffloadBundleURI>> UriOrErr(
+ OffloadBundleURI::createOffloadBundleURI(URIstr, FILE_URI));
+ if (!UriOrErr)
+ return UriOrErr.takeError();
+
+ OffloadBundleURI &Uri = **UriOrErr;
+ std::string OutputFile = Uri.FileName.str();
+ OutputFile +=
+ "-offset" + itostr(Uri.Offset) + "-size" + itostr(Uri.Size) + ".co";
+
+ // Create an ObjectFile object from uri.file_uri
+ auto ObjOrErr = ObjectFile::createObjectFile(Uri.FileName);
+ if (!ObjOrErr)
+ return ObjOrErr.takeError();
+
+ auto Obj = ObjOrErr->getBinary();
+ if (Error Err =
+ object::extractCodeObject(*Obj, Uri.Offset, Uri.Size, OutputFile))
+ return Err;
+
+ return Error::success();
+}
+
+// Utility function to format numbers with commas
+static std::string formatWithCommas(unsigned long long Value) {
+ std::string Num = std::to_string(Value);
+ int InsertPosition = Num.length() - 3;
+ while (InsertPosition > 0) {
+ Num.insert(InsertPosition, ",");
+ InsertPosition -= 3;
+ }
+ return Num;
+}
+
+llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
+CompressedOffloadBundle::decompress(llvm::MemoryBufferRef &Input,
+ bool Verbose) {
+ StringRef Blob = Input.getBuffer();
+
+ if (Blob.size() < V1HeaderSize)
+ return llvm::MemoryBuffer::getMemBufferCopy(Blob);
+
+ if (llvm::identify_magic(Blob) !=
+ llvm::file_magic::offload_bundle_compressed) {
+ if (Verbose)
+ llvm::errs() << "Uncompressed bundle.\n";
+ return llvm::MemoryBuffer::getMemBufferCopy(Blob);
+ }
+
+ size_t CurrentOffset = MagicSize;
+
+ uint16_t ThisVersion;
+ memcpy(&ThisVersion, Blob.data() + CurrentOffset, sizeof(uint16_t));
+ CurrentOffset += VersionFieldSize;
+
+ uint16_t CompressionMethod;
+ memcpy(&CompressionMethod, Blob.data() + CurrentOffset, sizeof(uint16_t));
+ CurrentOffset += MethodFieldSize;
+
+ uint32_t TotalFileSize;
+ if (ThisVersion >= 2) {
+ if (Blob.size() < V2HeaderSize)
+ return createStringError(inconvertibleErrorCode(),
+ "Compressed bundle header size too small");
+ memcpy(&TotalFileSize, Blob.data() + CurrentOffset, sizeof(uint32_t));
+ CurrentOffset += FileSizeFieldSize;
+ }
+
+ uint32_t UncompressedSize;
+ memcpy(&UncompressedSize, Blob.data() + CurrentOffset, sizeof(uint32_t));
+ CurrentOffset += UncompressedSizeFieldSize;
+
+ uint64_t StoredHash;
+ memcpy(&StoredHash, Blob.data() + CurrentOffset, sizeof(uint64_t));
+ CurrentOffset += HashFieldSize;
+
+ llvm::compression::Format CompressionFormat;
+ if (CompressionMethod ==
+ static_cast<uint16_t>(llvm::compression::Format::Zlib))
+ CompressionFormat = llvm::compression::Format::Zlib;
+ else if (CompressionMethod ==
+ static_cast<uint16_t>(llvm::compression::Format::Zstd))
+ CompressionFormat = llvm::compression::Format::Zstd;
+ else
+ return createStringError(inconvertibleErrorCode(),
+ "Unknown compressing method");
+
+ llvm::Timer DecompressTimer("Decompression Timer", "Decompression time",
+ OffloadBundlerTimerGroup);
+ if (Verbose)
+ DecompressTime...
[truncated]
|
david-salinas
force-pushed
the
extend-llvm-objdump-fatbin
branch
from
71a9eb1 to
228b0b2
Compare
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
extend option --offloading Change-Id: Ibc865f80e30aa1a6e5495ecfe617be68a5e15fcf
david-salinas
force-pushed
the
extend-llvm-objdump-fatbin
branch
from
228b0b2 to
13b5c8e
Compare
jhuber6
approved these changes
May 22, 2025
|
Hi, were seeing test failures from this patch in our CI. Error: ******************** TEST 'LLVM :: tools/llvm-objdump/Offloading/fatbin.test' FAILED ********************
Exit Code: 1
Command Output (stdout):
--
/b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf: file format elf64-x86-64
Extracting offload bundle: /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.host-x86_64-unknown-linux--
Extracting offload bundle: /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.hipv4-amdgcn-amd-amdhsa--gfx908
--
Command Output (stderr):
--
/b/s/w/ir/x/w/llvm_build/bin/yaml2obj /b/s/w/ir/x/w/llvm-llvm-project/llvm/test/tools/llvm-objdump/Offloading/fatbin.test -o /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf # RUN: at line 4
+ /b/s/w/ir/x/w/llvm_build/bin/yaml2obj /b/s/w/ir/x/w/llvm-llvm-project/llvm/test/tools/llvm-objdump/Offloading/fatbin.test -o /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf
/b/s/w/ir/x/w/llvm_build/bin/llvm-objdump --offloading /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf # RUN: at line 5
+ /b/s/w/ir/x/w/llvm_build/bin/llvm-objdump --offloading /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf
/b/s/w/ir/x/w/llvm_build/bin/llvm-objdump -d /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.hipv4-amdgcn-amd-amdhsa--gfx908 | /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/llvm/test/tools/llvm-objdump/Offloading/fatbin.test # RUN: at line 6
+ /b/s/w/ir/x/w/llvm_build/bin/llvm-objdump -d /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.hipv4-amdgcn-amd-amdhsa--gfx908
+ /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/llvm/test/tools/llvm-objdump/Offloading/fatbin.test
/b/s/w/ir/x/w/llvm_build/bin/llvm-objdump: error: '/b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.hipv4-amdgcn-amd-amdhsa--gfx908': can't find target: unable to get target for 'amdgcn-amd-amdhsa', see --version and --triple.
/b/s/w/ir/x/w/llvm-llvm-project/llvm/test/tools/llvm-objdump/Offloading/fatbin.test:9:10: error: CHECK: expected string not found in input
# CHECK: s_load_dword s7, s[4:5], 0x24
^
<stdin>:1:1: note: scanning from here
^
<stdin>:2:51: note: possible intended match here
/b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.hipv4-amdgcn-amd-amdhsa--gfx908: file format elf64-amdgpu
^
Input file: <stdin>
Check file: /b/s/w/ir/x/w/llvm-llvm-project/llvm/test/tools/llvm-objdump/Offloading/fatbin.test
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1:
check:9'0 X error: no match found
2: /b/s/w/ir/x/w/llvm_build/test/tools/llvm-objdump/Offloading/Output/fatbin.test.tmp.elf.0.hipv4-amdgcn-amd-amdhsa--gfx908: file format elf64-amdgpu
check:9'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:9'1 ? possible intended match
>>>>>>
--
********************If it isn't trivial to fix, would you mind reverting until a fixed version is ready? |
hi I believe it is a trivial fix. I'll post a fix asap. |
|
Thanks! |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/11/builds/15785 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/133/builds/16647 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/154/builds/16494 Here is the relevant piece of the build log for the reference |
|
@david-salinas @jhuber6 I've landed 3e3b02f to fix warnings from this PR. Thanks! |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/3/builds/16405 Here is the relevant piece of the build log for the reference |
|
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/27/builds/10487 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/168/builds/12406 Here is the relevant piece of the build log for the reference |
Thanks! |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/64/builds/3766 Here is the relevant piece of the build log for the reference |
|
This change has introduces crashes in unit tests on 32-bit x86: |
searlmc1
pushed a commit
to ROCm/llvm-project
that referenced
this pull request
May 24, 2025
This reverts commit 51a03ed.
searlmc1
pushed a commit
to ROCm/llvm-project
that referenced
this pull request
May 26, 2025
Utilize the new extensions to the LLVM Offloading API to extend to llvm-objdump to handle dumping fatbin offload bundles generated by HIP. This extension to llvm-objdump adds the option --offload-fatbin. Specifying this option will take the input object/executable and extract all offload fatbin bundle entries into distinct code object files with names reflecting the source file name combined with the Bundle Entry ID. Users can also use the --arch-name option to filter offload fatbin bundle entries by their target triple. --------- Co-authored-by: dsalinas <[email protected]>
searlmc1
pushed a commit
to ROCm/llvm-project
that referenced
this pull request
May 26, 2025
|
@david-salinas Are you looking at these test failures? |
Hi @david-salinas This change has disabled all 32 - bit arm buildbots into failure state since last 4 days. Kindly revert the change for now so that bots can start reporting again. |
DavidSpickett
added a commit
to DavidSpickett/llvm-project
that referenced
this pull request
May 27, 2025
Which fixes a test failure seen on the bots, introduced by llvm#140286. [ RUN ] OffloadingBundleTest.checkExtractOffloadBundleFatBinary ObjectTests: ../llvm/llvm/include/llvm/ADT/StringRef.h:618: StringRef llvm::StringRef::drop_front(size_t) const: Assertion `size() >= N && "Dropping more elements than exist"' failed. 0 0x0a24a990 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x31a990) 1 0x0a248364 llvm::sys::RunSignalHandlers() (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x318364) 2 0x0a24b410 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0 3 0xf46ed6f0 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0 4 0xf46ddb06 ./csu/../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47:0 5 0xf471d292 __pthread_kill_implementation ./nptl/pthread_kill.c:44:76 6 0xf46ec840 gsignal ./signal/../sysdeps/posix/raise.c:27:6 Also reported on 32-bit x86. The cause is that the code was casting the result of StringRef.find into an int64_t. The failure value of find is StringRef::npos, which is defined as: static constexpr size_t npos = ~size_t(0); So making that into an int64_t means it is > 0 and the code continued to search for bundles that didn't exist, at an offset beyond the largest file we could handle. I have changed the code to use size_t throughout, as this matches the APIs used, and it does not search backwards in the file so the offset does not need to be signed.
DavidSpickett
added a commit
that referenced
this pull request
May 28, 2025
Which fixes a test failure seen on the bots, introduced by #140286. ``` [ RUN ] OffloadingBundleTest.checkExtractOffloadBundleFatBinary ObjectTests: ../llvm/llvm/include/llvm/ADT/StringRef.h:618: StringRef llvm::StringRef::drop_front(size_t) const: Assertion `size() >= N && "Dropping more elements than exist"' failed. 0 0x0a24a990 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x31a990) 1 0x0a248364 llvm::sys::RunSignalHandlers() (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x318364) 2 0x0a24b410 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0 3 0xf46ed6f0 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0 4 0xf46ddb06 ./csu/../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47:0 5 0xf471d292 __pthread_kill_implementation ./nptl/pthread_kill.c:44:76 6 0xf46ec840 gsignal ./signal/../sysdeps/posix/raise.c:27:6 ``` Also reported on 32-bit x86. I think the cause is the code was casting the result of StringRef.find into an int64_t. The failure value of find is StringRef::npos, which is defined as: static constexpr size_t npos = ~size_t(0); * size_t(0) is 32 bits of 0s * the inverse of that is 32 bits of 1s * Cast to int64_t needs to widen this, and it will preserve the original value in doing so, which is 0xffffffff. * The result is 0x00000000ffffffff, which is >= 0, so we keep searching and try to go off the end of the file. Or put another way, this equivalent function returns true when compiled for a 32-bit system: ``` bool fn() { size_t foo = ~size_t(0); int64_t foo64 = (int64_t)foo; return foo64 >= 0; } ``` Using size_t throughout fixes the problem. Also I don't see a reason it needs to be a signed number, given that it always searches forward from the current offset.
llvm-sync bot
pushed a commit
to arm/arm-toolchain
that referenced
this pull request
May 28, 2025
… (#141620) Which fixes a test failure seen on the bots, introduced by llvm/llvm-project#140286. ``` [ RUN ] OffloadingBundleTest.checkExtractOffloadBundleFatBinary ObjectTests: ../llvm/llvm/include/llvm/ADT/StringRef.h:618: StringRef llvm::StringRef::drop_front(size_t) const: Assertion `size() >= N && "Dropping more elements than exist"' failed. 0 0x0a24a990 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x31a990) 1 0x0a248364 llvm::sys::RunSignalHandlers() (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x318364) 2 0x0a24b410 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0 3 0xf46ed6f0 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0 4 0xf46ddb06 ./csu/../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47:0 5 0xf471d292 __pthread_kill_implementation ./nptl/pthread_kill.c:44:76 6 0xf46ec840 gsignal ./signal/../sysdeps/posix/raise.c:27:6 ``` Also reported on 32-bit x86. I think the cause is the code was casting the result of StringRef.find into an int64_t. The failure value of find is StringRef::npos, which is defined as: static constexpr size_t npos = ~size_t(0); * size_t(0) is 32 bits of 0s * the inverse of that is 32 bits of 1s * Cast to int64_t needs to widen this, and it will preserve the original value in doing so, which is 0xffffffff. * The result is 0x00000000ffffffff, which is >= 0, so we keep searching and try to go off the end of the file. Or put another way, this equivalent function returns true when compiled for a 32-bit system: ``` bool fn() { size_t foo = ~size_t(0); int64_t foo64 = (int64_t)foo; return foo64 >= 0; } ``` Using size_t throughout fixes the problem. Also I don't see a reason it needs to be a signed number, given that it always searches forward from the current offset.
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/39/builds/6269 Here is the relevant piece of the build log for the reference |
|
That failure is the one I fixed, that bot is very far behind. |
sivan-shani
pushed a commit
to sivan-shani/llvm-project
that referenced
this pull request
Jun 3, 2025
Utilize the new extensions to the LLVM Offloading API to extend to llvm-objdump to handle dumping fatbin offload bundles generated by HIP. This extension to llvm-objdump adds the option --offload-fatbin. Specifying this option will take the input object/executable and extract all offload fatbin bundle entries into distinct code object files with names reflecting the source file name combined with the Bundle Entry ID. Users can also use the --arch-name option to filter offload fatbin bundle entries by their target triple. --------- Co-authored-by: dsalinas <[email protected]>
sivan-shani
pushed a commit
to sivan-shani/llvm-project
that referenced
this pull request
Jun 3, 2025
Which fixes a test failure seen on the bots, introduced by llvm#140286. ``` [ RUN ] OffloadingBundleTest.checkExtractOffloadBundleFatBinary ObjectTests: ../llvm/llvm/include/llvm/ADT/StringRef.h:618: StringRef llvm::StringRef::drop_front(size_t) const: Assertion `size() >= N && "Dropping more elements than exist"' failed. 0 0x0a24a990 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x31a990) 1 0x0a248364 llvm::sys::RunSignalHandlers() (/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/unittests/Object/./ObjectTests+0x318364) 2 0x0a24b410 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0 3 0xf46ed6f0 __default_rt_sa_restorer ./signal/../sysdeps/unix/sysv/linux/arm/sigrestorer.S:80:0 4 0xf46ddb06 ./csu/../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47:0 5 0xf471d292 __pthread_kill_implementation ./nptl/pthread_kill.c:44:76 6 0xf46ec840 gsignal ./signal/../sysdeps/posix/raise.c:27:6 ``` Also reported on 32-bit x86. I think the cause is the code was casting the result of StringRef.find into an int64_t. The failure value of find is StringRef::npos, which is defined as: static constexpr size_t npos = ~size_t(0); * size_t(0) is 32 bits of 0s * the inverse of that is 32 bits of 1s * Cast to int64_t needs to widen this, and it will preserve the original value in doing so, which is 0xffffffff. * The result is 0x00000000ffffffff, which is >= 0, so we keep searching and try to go off the end of the file. Or put another way, this equivalent function returns true when compiled for a 32-bit system: ``` bool fn() { size_t foo = ~size_t(0); int64_t foo64 = (int64_t)foo; return foo64 >= 0; } ``` Using size_t throughout fixes the problem. Also I don't see a reason it needs to be a signed number, given that it always searches forward from the current offset.
jrbyrnes
pushed a commit
to jrbyrnes/llvm-project
that referenced
this pull request
Jun 6, 2025
Utilize the new extensions to the LLVM Offloading API to extend to llvm-objdump to handle dumping fatbin offload bundles generated by HIP. This extension to llvm-objdump adds the option --offload-fatbin. Specifying this option will take the input object/executable and extract all offload fatbin bundle entries into distinct code object files with names reflecting the source file name combined with the Bundle Entry ID. Users can also use the --arch-name option to filter offload fatbin bundle entries by their target triple. --------- Co-authored-by: dsalinas <[email protected]>
jrbyrnes
pushed a commit
to jrbyrnes/llvm-project
that referenced
this pull request
Jun 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Utilize the new extensions to the LLVM Offloading API to extend to llvm-objdump to handle dumping fatbin offload bundles generated by HIP. This extension to llvm-objdump adds the option --offload-fatbin. Specifying this option will take the input object/executable and extract all offload fatbin bundle entries into distinct code object files with names reflecting the source file name combined with the Bundle Entry ID. Users can also use the --arch-name option to filter offload fatbin bundle entries by their target triple.