[JITLink][XCOFF] Setup initial build support for XCOFF #127266
Conversation
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
|
Looks great so far! Do you have a timeline in mind for supporting a minimal program like: int main(int argc, char *argv[]) {
return 0;
}? |
So far I have a prototype of For PPC64, we have already implemented a large portion of the relocation fixup code for ELF that we can use for XCOFF as well. Implmenting the linking portion for XCOFF should not take much more than another week or 2. Which should allow us to compile simple programs. |
Sounds great. I just tried the following: % cat main-ret-zero.c
int main(int argc, char *argv[]) {
return 0;
}
% clang --target=powerpc64-ibm-aix-xcoff -fPIC -c -o main-ret-zero.o main-ret-zero.c
% llvm-objdump -r main-ret-zero.o
main-ret-zero.o: file format aix5coff64-rs6000
RELOCATION RECORDS FOR [.data]:
OFFSET TYPE VALUE
0000000000000000 R_POS .main
0000000000000008 R_POS TOCSo it looks like you only need support for
Very cool! |
|
I completed the work required to run a simple main with an immediate return code. I will clean up the patch and post it soon. But I ran into 2 problems that I need some suggestions for: First, in order to call the main function, we need to be able to indirectly call through the function descriptor So the symbol JITSymbolFlag::None for main and JITSymbolFlag::Callable only.
Second, |
The initial JITSymbolFlags values should be created as part of object file interface construction here, with flags for individual symbols within the interface computed here. In particular I would expect these lines to set the It might be worth stepping through a test case in the debugger to see what interface is produced.
We can't use C test cases as llvm tests can't assume that clang is available (e.g. someone might be doing an LLVM-only build on a system with gcc as the only available compiler). You can use obj2yaml to produce a yaml representation of your object if that helps, but if MC doesn't work then I suspect obj2yaml might also have issues. Binary object files are allowable as a last resort.
That seems fine to me as a test case for the initial checkin, except that it'll need to be an assembly, yaml or binary test case rather than C. |
mustartt
force-pushed
the
jitlink-xcoff
branch
from
968039c to
c11b614
Compare
|
@llvm/pr-subscribers-llvm-binary-utilities Author: Henry Jiang (mustartt) ChangesThis patch starts the initial implementation of JITLink for XCOFF (Object format for AIX). Patch is 34.86 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/127266.diff 12 Files Affected:
diff --git a/llvm/include/llvm/ExecutionEngine/JITLink/XCOFF.h b/llvm/include/llvm/ExecutionEngine/JITLink/XCOFF.h
new file mode 100644
index 0000000000000..3d181d0786eb7
--- /dev/null
+++ b/llvm/include/llvm/ExecutionEngine/JITLink/XCOFF.h
@@ -0,0 +1,37 @@
+//===------- XCOFF.h - Generic JIT link function for XCOFF ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// jit-link functions for XCOFF.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_JITLINK_XCOFF_H
+#define LLVM_EXECUTIONENGINE_JITLINK_XCOFF_H
+
+#include "llvm/ExecutionEngine/JITLink/JITLink.h"
+
+namespace llvm {
+namespace jitlink {
+
+/// Create a LinkGraph from an XCOFF relocatable object.
+///
+/// Note: The graph does not take ownership of the underlying buffer, nor copy
+/// its contents. The caller is responsible for ensuring that the object buffer
+/// outlives the graph.
+Expected<std::unique_ptr<LinkGraph>>
+createLinkGraphFromXCOFFObject(MemoryBufferRef ObjectBuffer,
+ std::shared_ptr<orc::SymbolStringPool> SSP);
+
+/// Link the given graph.
+void link_XCOFF(std::unique_ptr<LinkGraph> G,
+ std::unique_ptr<JITLinkContext> Ctx);
+
+} // namespace jitlink
+} // namespace llvm
+
+#endif // LLVM_EXECUTIONENGINE_JITLINK_XCOFF_H
diff --git a/llvm/include/llvm/ExecutionEngine/JITLink/XCOFF_ppc64.h b/llvm/include/llvm/ExecutionEngine/JITLink/XCOFF_ppc64.h
new file mode 100644
index 0000000000000..ec5c8a37bda27
--- /dev/null
+++ b/llvm/include/llvm/ExecutionEngine/JITLink/XCOFF_ppc64.h
@@ -0,0 +1,37 @@
+//===------ XCOFF_ppc64.h - JIT link functions for XCOFF/ppc64 ------*- C++
+//-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// jit-link functions for XCOFF/ppc64.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_JITLINK_XCOFF_PPC64_H
+#define LLVM_EXECUTIONENGINE_JITLINK_XCOFF_PPC64_H
+
+#include "llvm/ExecutionEngine/JITLink/JITLink.h"
+
+namespace llvm::jitlink {
+
+/// Create a LinkGraph from an XCOFF/ppc64 relocatable object.
+///
+/// Note: The graph does not take ownership of the underlying buffer, nor copy
+/// its contents. The caller is responsible for ensuring that the object buffer
+/// outlives the graph.
+///
+Expected<std::unique_ptr<LinkGraph>> createLinkGraphFromXCOFFObject_ppc64(
+ MemoryBufferRef ObjectBuffer, std::shared_ptr<orc::SymbolStringPool> SSP);
+
+/// jit-link the given object buffer, which must be a XCOFF ppc64 object file.
+///
+void link_XCOFF_ppc64(std::unique_ptr<LinkGraph> G,
+ std::unique_ptr<JITLinkContext> Ctx);
+
+} // end namespace llvm::jitlink
+
+#endif // LLVM_EXECUTIONENGINE_JITLINK_XCOFF_PPC64_H
diff --git a/llvm/lib/ExecutionEngine/JITLink/CMakeLists.txt b/llvm/lib/ExecutionEngine/JITLink/CMakeLists.txt
index 65dd0c7468ae1..22e4513e1374c 100644
--- a/llvm/lib/ExecutionEngine/JITLink/CMakeLists.txt
+++ b/llvm/lib/ExecutionEngine/JITLink/CMakeLists.txt
@@ -35,6 +35,11 @@ add_llvm_component_library(LLVMJITLink
COFFLinkGraphBuilder.cpp
COFF_x86_64.cpp
+ # XCOFF
+ XCOFF.cpp
+ XCOFF_ppc64.cpp
+ XCOFFLinkGraphBuilder.cpp
+
# Architectures:
aarch32.cpp
aarch64.cpp
diff --git a/llvm/lib/ExecutionEngine/JITLink/JITLink.cpp b/llvm/lib/ExecutionEngine/JITLink/JITLink.cpp
index e8ce9b2b9527d..15a8fcf312ade 100644
--- a/llvm/lib/ExecutionEngine/JITLink/JITLink.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/JITLink.cpp
@@ -13,6 +13,7 @@
#include "llvm/ExecutionEngine/JITLink/COFF.h"
#include "llvm/ExecutionEngine/JITLink/ELF.h"
#include "llvm/ExecutionEngine/JITLink/MachO.h"
+#include "llvm/ExecutionEngine/JITLink/XCOFF.h"
#include "llvm/ExecutionEngine/JITLink/aarch64.h"
#include "llvm/ExecutionEngine/JITLink/i386.h"
#include "llvm/ExecutionEngine/JITLink/loongarch.h"
@@ -501,6 +502,8 @@ createLinkGraphFromObject(MemoryBufferRef ObjectBuffer,
return createLinkGraphFromELFObject(ObjectBuffer, std::move(SSP));
case file_magic::coff_object:
return createLinkGraphFromCOFFObject(ObjectBuffer, std::move(SSP));
+ case file_magic::xcoff_object_64:
+ return createLinkGraphFromXCOFFObject(ObjectBuffer, std::move(SSP));
default:
return make_error<JITLinkError>("Unsupported file format");
};
@@ -532,6 +535,8 @@ void link(std::unique_ptr<LinkGraph> G, std::unique_ptr<JITLinkContext> Ctx) {
return link_ELF(std::move(G), std::move(Ctx));
case Triple::COFF:
return link_COFF(std::move(G), std::move(Ctx));
+ case Triple::XCOFF:
+ return link_XCOFF(std::move(G), std::move(Ctx));
default:
Ctx->notifyFailed(make_error<JITLinkError>("Unsupported object format"));
};
diff --git a/llvm/lib/ExecutionEngine/JITLink/XCOFF.cpp b/llvm/lib/ExecutionEngine/JITLink/XCOFF.cpp
new file mode 100644
index 0000000000000..cb026538632a9
--- /dev/null
+++ b/llvm/lib/ExecutionEngine/JITLink/XCOFF.cpp
@@ -0,0 +1,43 @@
+//===-------------- XCOFF.cpp - JIT linker function for XCOFF -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// XCOFF jit-link function.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ExecutionEngine/JITLink/XCOFF.h"
+#include "llvm/ExecutionEngine/JITLink/XCOFF_ppc64.h"
+#include "llvm/Object/XCOFFObjectFile.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "jitlink"
+
+namespace llvm {
+namespace jitlink {
+
+Expected<std::unique_ptr<LinkGraph>>
+createLinkGraphFromXCOFFObject(MemoryBufferRef ObjectBuffer,
+ std::shared_ptr<orc::SymbolStringPool> SSP) {
+ // Check magic
+ file_magic Magic = identify_magic(ObjectBuffer.getBuffer());
+ if (Magic != file_magic::xcoff_object_64)
+ return make_error<JITLinkError>("Invalid XCOFF 64 Header");
+
+ // TODO: See if we need to add more checks
+ //
+ return createLinkGraphFromXCOFFObject_ppc64(ObjectBuffer, std::move(SSP));
+}
+
+void link_XCOFF(std::unique_ptr<LinkGraph> G,
+ std::unique_ptr<JITLinkContext> Ctx) {
+ link_XCOFF_ppc64(std::move(G), std::move(Ctx));
+}
+
+} // namespace jitlink
+} // namespace llvm
diff --git a/llvm/lib/ExecutionEngine/JITLink/XCOFFLinkGraphBuilder.cpp b/llvm/lib/ExecutionEngine/JITLink/XCOFFLinkGraphBuilder.cpp
new file mode 100644
index 0000000000000..0a76dcc77536b
--- /dev/null
+++ b/llvm/lib/ExecutionEngine/JITLink/XCOFFLinkGraphBuilder.cpp
@@ -0,0 +1,420 @@
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Generic XCOFF LinkGraph building code.
+//
+//===----------------------------------------------------------------------===//
+
+#include "XCOFFLinkGraphBuilder.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/BinaryFormat/XCOFF.h"
+#include "llvm/ExecutionEngine/JITLink/JITLink.h"
+#include "llvm/ExecutionEngine/JITLink/ppc64.h"
+#include "llvm/ExecutionEngine/Orc/Shared/ExecutorAddress.h"
+#include "llvm/ExecutionEngine/Orc/Shared/MemoryFlags.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/Object/XCOFFObjectFile.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/Format.h"
+#include "llvm/Support/raw_ostream.h"
+#include <memory>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "jitlink"
+
+namespace llvm {
+namespace jitlink {
+
+XCOFFLinkGraphBuilder::XCOFFLinkGraphBuilder(
+ const object::XCOFFObjectFile &Obj,
+ std::shared_ptr<orc::SymbolStringPool> SSP, Triple TT,
+ SubtargetFeatures Features,
+ LinkGraph::GetEdgeKindNameFunction GetEdgeKindName)
+ : Obj(Obj),
+ G(std::make_unique<LinkGraph>(
+ std::string(Obj.getFileName()), std::move(SSP), std::move(TT),
+ std::move(Features), std::move(GetEdgeKindName))) {}
+
+static llvm::StringRef getStorageClassString(XCOFF::StorageClass SC) {
+ switch (SC) {
+ case XCOFF::StorageClass::C_FILE:
+ return "C_FILE (File name)";
+ case XCOFF::StorageClass::C_BINCL:
+ return "C_BINCL (Beginning of include file)";
+ case XCOFF::StorageClass::C_EINCL:
+ return "C_EINCL (Ending of include file)";
+ case XCOFF::StorageClass::C_GSYM:
+ return "C_GSYM (Global variable)";
+ case XCOFF::StorageClass::C_STSYM:
+ return "C_STSYM (Statically allocated symbol)";
+ case XCOFF::StorageClass::C_BCOMM:
+ return "C_BCOMM (Beginning of common block)";
+ case XCOFF::StorageClass::C_ECOMM:
+ return "C_ECOMM (End of common block)";
+ case XCOFF::StorageClass::C_ENTRY:
+ return "C_ENTRY (Alternate entry)";
+ case XCOFF::StorageClass::C_BSTAT:
+ return "C_BSTAT (Beginning of static block)";
+ case XCOFF::StorageClass::C_ESTAT:
+ return "C_ESTAT (End of static block)";
+ case XCOFF::StorageClass::C_GTLS:
+ return "C_GTLS (Global thread-local variable)";
+ case XCOFF::StorageClass::C_STTLS:
+ return "C_STTLS (Static thread-local variable)";
+ case XCOFF::StorageClass::C_DWARF:
+ return "C_DWARF (DWARF section symbol)";
+ case XCOFF::StorageClass::C_LSYM:
+ return "C_LSYM (Automatic variable allocated on stack)";
+ case XCOFF::StorageClass::C_PSYM:
+ return "C_PSYM (Argument to subroutine allocated on stack)";
+ case XCOFF::StorageClass::C_RSYM:
+ return "C_RSYM (Register variable)";
+ case XCOFF::StorageClass::C_RPSYM:
+ return "C_RPSYM (Argument to function stored in register)";
+ case XCOFF::StorageClass::C_ECOML:
+ return "C_ECOML (Local member of common block)";
+ case XCOFF::StorageClass::C_FUN:
+ return "C_FUN (Function or procedure)";
+ case XCOFF::StorageClass::C_EXT:
+ return "C_EXT (External symbol)";
+ case XCOFF::StorageClass::C_WEAKEXT:
+ return "C_WEAKEXT (Weak external symbol)";
+ case XCOFF::StorageClass::C_NULL:
+ return "C_NULL";
+ case XCOFF::StorageClass::C_STAT:
+ return "C_STAT (Static)";
+ case XCOFF::StorageClass::C_BLOCK:
+ return "C_BLOCK (\".bb\" or \".eb\")";
+ case XCOFF::StorageClass::C_FCN:
+ return "C_FCN (\".bf\" or \".ef\")";
+ case XCOFF::StorageClass::C_HIDEXT:
+ return "C_HIDEXT (Un-named external symbol)";
+ case XCOFF::StorageClass::C_INFO:
+ return "C_INFO (Comment string in .info section)";
+ case XCOFF::StorageClass::C_DECL:
+ return "C_DECL (Declaration of object)";
+ case XCOFF::StorageClass::C_AUTO:
+ return "C_AUTO (Automatic variable)";
+ case XCOFF::StorageClass::C_REG:
+ return "C_REG (Register variable)";
+ case XCOFF::StorageClass::C_EXTDEF:
+ return "C_EXTDEF (External definition)";
+ case XCOFF::StorageClass::C_LABEL:
+ return "C_LABEL (Label)";
+ case XCOFF::StorageClass::C_ULABEL:
+ return "C_ULABEL (Undefined label)";
+ case XCOFF::StorageClass::C_MOS:
+ return "C_MOS (Member of structure)";
+ case XCOFF::StorageClass::C_ARG:
+ return "C_ARG (Function argument)";
+ case XCOFF::StorageClass::C_STRTAG:
+ return "C_STRTAG (Structure tag)";
+ case XCOFF::StorageClass::C_MOU:
+ return "C_MOU (Member of union)";
+ case XCOFF::StorageClass::C_UNTAG:
+ return "C_UNTAG (Union tag)";
+ case XCOFF::StorageClass::C_TPDEF:
+ return "C_TPDEF (Type definition)";
+ case XCOFF::StorageClass::C_USTATIC:
+ return "C_USTATIC (Undefined static)";
+ case XCOFF::StorageClass::C_ENTAG:
+ return "C_ENTAG (Enumeration tag)";
+ case XCOFF::StorageClass::C_MOE:
+ return "C_MOE (Member of enumeration)";
+ case XCOFF::StorageClass::C_REGPARM:
+ return "C_REGPARM (Register parameter)";
+ case XCOFF::StorageClass::C_FIELD:
+ return "C_FIELD (Bit field)";
+ case XCOFF::StorageClass::C_EOS:
+ return "C_EOS (End of structure)";
+ case XCOFF::StorageClass::C_LINE:
+ return "C_LINE";
+ case XCOFF::StorageClass::C_ALIAS:
+ return "C_ALIAS (Duplicate tag)";
+ case XCOFF::StorageClass::C_HIDDEN:
+ return "C_HIDDEN (Special storage class for external)";
+ case XCOFF::StorageClass::C_EFCN:
+ return "C_EFCN (Physical end of function)";
+ case XCOFF::StorageClass::C_TCSYM:
+ return "C_TCSYM (Reserved)";
+ }
+}
+
+Error XCOFFLinkGraphBuilder::processSections() {
+ LLVM_DEBUG(dbgs() << " Creating graph sections...\n");
+
+ UndefSection = &G->createSection("*UND*", orc::MemProt::None);
+
+ for (object::SectionRef Section : Obj.sections()) {
+ auto SectionName = Section.getName();
+ if (!SectionName)
+ return SectionName.takeError();
+
+ LLVM_DEBUG({
+ dbgs() << " section = " << *SectionName
+ << ", idx = " << Section.getIndex()
+ << ", size = " << format_hex_no_prefix(Section.getSize(), 8)
+ << ", vma = " << format_hex(Section.getAddress(), 16) << "\n";
+ });
+
+ // We can skip debug (including dawrf) and pad sections
+ if (Section.isDebugSection() || *SectionName == "pad")
+ continue;
+ LLVM_DEBUG(dbgs() << " creating graph section\n");
+
+ orc::MemProt Prot = orc::MemProt::Read;
+ if (Section.isText())
+ Prot |= orc::MemProt::Exec;
+ if (Section.isData() || Section.isBSS())
+ Prot |= orc::MemProt::Write;
+
+ jitlink::Section *GraphSec = &G->createSection(*SectionName, Prot);
+ // TODO: Check for no_alloc for certain sections
+
+ assert(!SectionTable.contains(Section.getIndex()) &&
+ "Section with same index already exists");
+ SectionTable[Section.getIndex()] = {GraphSec, Section};
+ }
+
+ return Error::success();
+}
+
+static std::optional<object::XCOFFSymbolRef>
+getXCOFFSymbolContainingSymbolRef(const object::XCOFFObjectFile &Obj,
+ const object::SymbolRef &Sym) {
+ const object::XCOFFSymbolRef SymRef =
+ Obj.toSymbolRef(Sym.getRawDataRefImpl());
+ if (!SymRef.isCsectSymbol())
+ return std::nullopt;
+
+ Expected<object::XCOFFCsectAuxRef> CsectAuxEntOrErr =
+ SymRef.getXCOFFCsectAuxRef();
+ if (!CsectAuxEntOrErr || !CsectAuxEntOrErr.get().isLabel())
+ return std::nullopt;
+ uint32_t Idx =
+ static_cast<uint32_t>(CsectAuxEntOrErr.get().getSectionOrLength());
+ object::DataRefImpl DRI;
+ DRI.p = Obj.getSymbolByIndex(Idx);
+ return object::XCOFFSymbolRef(DRI, &Obj);
+}
+
+static void printSymbolEntry(raw_ostream &OS,
+ const object::XCOFFObjectFile &Obj,
+ const object::XCOFFSymbolRef &Sym) {
+ OS << " " << format_hex(cantFail(Sym.getAddress()), 16);
+ OS << " " << left_justify(cantFail(Sym.getName()), 10);
+ if (Sym.isCsectSymbol()) {
+ auto CsectAuxEntry = cantFail(Sym.getXCOFFCsectAuxRef());
+ if (!CsectAuxEntry.isLabel()) {
+ std::string MCStr =
+ "[" +
+ XCOFF::getMappingClassString(CsectAuxEntry.getStorageMappingClass())
+ .str() +
+ "]";
+ OS << left_justify(MCStr, 3);
+ }
+ }
+ OS << " " << format_hex(Sym.getSize(), 8);
+ OS << " " << Sym.getSectionNumber();
+ OS << " " << getStorageClassString(Sym.getStorageClass());
+ OS << " (idx: " << Obj.getSymbolIndex(Sym.getRawDataRefImpl().p) << ")";
+ if (Sym.isCsectSymbol()) {
+ if (auto ParentSym = getXCOFFSymbolContainingSymbolRef(Obj, Sym)) {
+ OS << " (csect idx: "
+ << Obj.getSymbolIndex(ParentSym->getRawDataRefImpl().p) << ")";
+ }
+ }
+ OS << "\n";
+}
+
+Error XCOFFLinkGraphBuilder::processCsectsAndSymbols() {
+ LLVM_DEBUG(dbgs() << " Creating graph blocks and symbols...\n");
+
+ for (auto [K, V] : SectionTable) {
+ LLVM_DEBUG(dbgs() << " section entry(idx: " << K
+ << " section: " << V.Section->getName() << ")\n");
+ }
+
+ for (object::XCOFFSymbolRef Symbol : Obj.symbols()) {
+ LLVM_DEBUG({ printSymbolEntry(dbgs(), Obj, Symbol); });
+
+ auto Flags = Symbol.getFlags();
+ if (!Flags)
+ return Flags.takeError();
+
+ bool External = *Flags & object::SymbolRef::SF_Undefined;
+ bool Weak = *Flags & object::SymbolRef::SF_Weak;
+ bool Hidden = *Flags & object::SymbolRef::SF_Hidden;
+ bool Global = *Flags & object::SymbolRef::SF_Global;
+
+ auto SymbolIndex = Obj.getSymbolIndex(Symbol.getEntryAddress());
+ auto SymbolName = Symbol.getName();
+ if (!SymbolName)
+ return SymbolName.takeError();
+
+ if (External) {
+ LLVM_DEBUG(dbgs() << " created external symbol\n");
+ SymbolIndexTable[SymbolIndex] =
+ &G->addExternalSymbol(*SymbolName, Symbol.getSize(), Weak);
+ continue;
+ }
+
+ if (!Symbol.isCsectSymbol()) {
+ LLVM_DEBUG(dbgs() << " skipped: not a csect symbol\n");
+ continue;
+ }
+
+ auto ParentSym = getXCOFFSymbolContainingSymbolRef(Obj, Symbol);
+ object::XCOFFSymbolRef CsectSymbol = ParentSym ? *ParentSym : Symbol;
+
+ auto CsectSymbolIndex = Obj.getSymbolIndex(CsectSymbol.getEntryAddress());
+ auto ParentSectionNumber = CsectSymbol.getSectionNumber();
+
+ bool IsUndefinedSection = !SectionTable.contains(ParentSectionNumber);
+ Section *ParentSection = !IsUndefinedSection
+ ? SectionTable[ParentSectionNumber].Section
+ : UndefSection;
+ Block *B = nullptr;
+
+ // TODO: Clean up the logic for handling undefined symbols
+ if (!CsectTable.contains(CsectSymbolIndex) && !IsUndefinedSection) {
+ object::SectionRef &SectionRef =
+ SectionTable[ParentSectionNumber].SectionData;
+ auto Data = SectionRef.getContents();
+ if (!Data)
+ return Data.takeError();
+ auto CsectSymbolAddr = CsectSymbol.getAddress();
+ if (!CsectSymbolAddr)
+ return CsectSymbolAddr.takeError();
+
+ ArrayRef<char> SectionBuffer{Data->data(), Data->size()};
+ auto Offset = *CsectSymbolAddr - SectionRef.getAddress();
+
+ LLVM_DEBUG(dbgs() << " symbol entry: offset = " << Offset
+ << ", size = " << CsectSymbol.getSize()
+ << ", storage class = "
+ << getStorageClassString(CsectSymbol.getStorageClass())
+ << "\n");
+
+ B = &G->createContentBlock(
+ *ParentSection, SectionBuffer.slice(Offset, CsectSymbol.getSize()),
+ orc::ExecutorAddr(*CsectSymbolAddr), CsectSymbol.getAlignment(), 0);
+
+ CsectTable[CsectSymbolIndex] = B;
+ } else {
+ B = CsectTable[CsectSymbolIndex];
+ }
+
+ Scope S{Scope::Default};
+ if (Hidden)
+ S = Scope::Hidden;
+ // TODO: Got this from llvm-objdump.cpp:2938 not sure if its correct
+ if (!Weak) {
+ if (Global)
+ S = Scope::Default;
+ else
+ S = Scope::Local;
+ }
+
+ // TODO: figure out all symbols that should have Scope::SideEffectsOnly
+ Linkage L = Weak ? Linkage::Weak : Linkage::Strong;
+ auto SymbolAddr = Symbol.getAddress();
+ if (!SymbolAddr)
+ return SymbolAddr.takeError();
+ auto IsCallableOrErr = Symbol.isFunction();
+ if (!IsCallableOrErr)
+ return IsCallableOrErr.takeError();
+
+ auto BlockOffset = *SymbolAddr - B->getAddress().getValue();
+
+ LLVM_DEBUG(dbgs() << " creating with linkage = " << getLinkageName(L)
+ << ", scope = " << getScopeName(S) << ", B = "
+ << format_hex(B->getAddress().getValue(), 16) << "\n");
+
+ SymbolIndexTable[SymbolIndex] =
+ &G->addDefinedSymbol(*B, BlockOffset, *SymbolName, Symbol.getSize(), L,
+ S, *IsCallableOrErr, true);
+ }
+
+ return Error::success();
+}
+
+Error XCOFFLinkGraphBuilder::processRelocations() {
+ LLVM_DEBUG(dbgs() << " Creating relocations...\n");
+
+ for (object::SectionRef Section : Obj.sections())...
[truncated]
|
|
Okay so I had the time to clean up the initial patch to support building simple programs. Here is my plan for the next following few patches:
|
| return SymFlags.takeError(); | ||
|
|
||
| // TODO: Global symbols should have default visibility for now | ||
| *SymFlags |= JITSymbolFlags::Exported; |
There was a problem hiding this comment.
Looks like XCOFFObjectFile::getSymbolFlags isn't returning SF_Exported for these symbols. The relevant logic from XCOFFObjectFile.cpp seems to be:
// There is no visibility in old 32 bit XCOFF object file interpret.
if (is64Bit() || (auxiliaryHeader32() && (auxiliaryHeader32()->getVersion() ==
NEW_XCOFF_INTERPRET))) {
uint16_t SymType = XCOFFSym.getSymbolType();
if ((SymType & VISIBILITY_MASK) == SYM_V_HIDDEN)
Result |= SymbolRef::SF_Hidden;
if ((SymType & VISIBILITY_MASK) == SYM_V_EXPORTED)
Result |= SymbolRef::SF_Exported;
}
return Result;Does that look right, or is it missing something?
There was a problem hiding this comment.
That does look correct. Need to verify with the xcoff experts to be sure.
There was a problem hiding this comment.
After verifying, added this to the comments
By default, only shared library with F_SHROBJ in the xcoff header will have SF_Exported. On AIX, we require the linker with the help of an export list to determine the visibility of the symbol. This mimics the behaviour of -bautoexp and CreateExportList utilities. In the abscence of the export list or
dlexportattribute in the IR, this is the best we can do for now.
https://www.ibm.com/docs/en/xl-c-aix/13.1.0?topic=library-exporting-symbols-createexportlist-utility
Its expected that given a XCOFF object, without going through the linker will not have SF_Exported flag set.
There was a problem hiding this comment.
It's questionable whether XCOFFObjectFile::getSymbolFlags is doing what is desired here. Firstly, the description of SF_Exported ("Symbol is visible to other DSOs") should apply to symbols with "protected" visibility. Secondly, if the input file has a loader section, the loader symbol table should be consulted (and if what the loader symbol table says is overridden here, there needs to be strong, well-articulated rationale).
There was a problem hiding this comment.
Based on https://www.ibm.com/docs/en/aix/7.3?topic=l-ld-command#ld__title__13, we should be exporting SF_Exported based on the ld documentation:
Protected The symbol is exported but cannot be rebounded (or preempted), even if runtime linking is being used.
Exported The symbol is exported with the global export attribute.
Will separate this change to another patch in case of test breaks.
There was a problem hiding this comment.
I agree that changing XCOFFObjectFile::getSymbolFlags should be separate from the current patch.
With respect to that patch (but off-topic for the current one), I want to emphasize that the getSymbolFlags implementation should ideally consult the loader symbol table when provided with an object that contains one. I do not know whether or not that has a practical effect for BOLT or other use cases.
| S = Scope::Local; | ||
| } | ||
|
|
||
| // TODO: figure out all symbols that should have Scope::SideEffectsOnly |
There was a problem hiding this comment.
SideEffectsOnly is special, and closely related to the initializer symbol (which should always be SideEffectsOnly):
A graph should have an initializer symbol if there are any static constructors in the graph (or related registrations, like ObjC / Swift metadata registrations on Darwin) that should run at dlopen time.
A side-effects only symbol is one that can be looked up to trigger linking of a graph, but which does not show up in the final symbol table (so it must always be looked up with WeaklyReferencedSymbol to avoid a SymbolsNotFound error). The initializer symbol should be marked with SideEffectsOnly, but API clients (e.g. plugins) can add their own special symbols if they want.
|
I'd love to resolve the |
Thanks for taking a look at the patch. The issue is that But for now I think it should be okay to land this if you are okay with it. I will fix this in the next patch or 2. |
|
I think the From the "Symbol Visibility" section in https://www.ibm.com/docs/en/aix/7.1?topic=l-ld-command:
So my question for the XCOFF experts is: What does |
From my understanding, C symbols will have
With AIX ld with the default option @hubert-reinterpretcast Does that sound correct? For JITLink, I think its reasonable to mimic what the linker would have done for object files linked with |
On AIX, symbols may have No Visibility.
It is the last case. Symbols with no visibility may be "exported" (but not given Exported visibility) via linker options (such as the use of an export file). |
As explained in my other response. The property of being exported is separate from the Visibility. Additionally, symbols with "protected" visibility are generally exported. Your understanding of |
Correction: For JITLink, I think its reasonable to mimic what the linker would have done for object files linked with This is the equivalent of linking with This is also a problem we are facing with the Rust drivers as well where we don't have access to the export list, and as a result we end up passing |
If you mean to consider XCOFF Aside: The document linked above seems rather silent on the topic of symbol interpositioning.
Off-topic, but that is likely going to run into trouble due to AIX's generally eager resolution behaviour. |
I am not familiar with the capabilities of the JIT linker. After (3), can we expect to be able to call (from a JIT-linked function) a function from the link target process that throws a C++ exception and be able to catch it in the JIT-linked function? I think that it would additionally make sense to observe how unwinding works for a similar scenario with C code. Also, are C++ static constructors (also applicable to C with the |
After 4, when we get all the relocation types used in eh_frames, working then we should be able to catch exceptions. Yes, C++ static initializers are on the roadmap. We need special registration for those symbols. |
From our offline discussion, you indicated that your primary interest in JITLink support is to support BOLT. I don't understand how BOLT uses JITLink, but it seems to me that the requirements associated with C++ static initializer support for JITLink (to collect them and arrange to run them without using |
It's in the roadmap but it is not the top priority. I'd like to support all the features and relocations implement first, and worry about things like the static initializers and such after. |
Can you give me an answer as to how JITLink fits into BOLT? I am trying to emphasize that the particular usage of JITLink for the purposes of BOLT can have additional complications in the context of AIX (which is not directly tied to XCOFF, but extends to additional linker behaviours). |
@lhames, has the clarification regarding the treatment of visibility in XCOFF been sufficient to confirm the proper direction for this patch? To summarize, it is technically "underspecified", but common (if not generally advisable) practice is to have these symbols be publicly visible if linked into a dynamic library (as opposed to a main executable). |
| return SymFlags.takeError(); | ||
|
|
||
| // TODO: Global symbols should have default visibility for now | ||
| *SymFlags |= JITSymbolFlags::Exported; |
There was a problem hiding this comment.
It's questionable whether XCOFFObjectFile::getSymbolFlags is doing what is desired here. Firstly, the description of SF_Exported ("Symbol is visible to other DSOs") should apply to symbols with "protected" visibility. Secondly, if the input file has a loader section, the loader symbol table should be consulted (and if what the loader symbol table says is overridden here, there needs to be strong, well-articulated rationale).
| if (!SymFlags) | ||
| return SymFlags.takeError(); | ||
|
|
||
| // By default, only shared library with F_SHROBJ in the xcoff header will |
There was a problem hiding this comment.
What do you mean by "default" here? What conditions count as "default" for this claim?
I can only give an high level overview. @scui-ibm would you mind giving a more in detailed summary? BOLT uses JITLink as an platform independent linker that has control over exactly how the sections are layed out in the final executable in order to minimize icache and dcache misses. After parsing the entire call graph and performing all the bolt related optimization, bolt will emit in memory object files for JITLink to resolve the relocations and reordering of the sections. |
|
Updated the rationale for keeping default visibility for |
I also only have high levels view of the interaction between BOLT and JITLinker. Basically after BOLT transformations, BOLT JITLinkLinker calls jitlink::createLinkGraphFromObject to create a LinkGraph and then calls jitlink::link to link it (section allocation and relocation fixups). There are some details/discussions in this PR (https://reviews.llvm.org/D147544, [BOLT] Move from RuntimeDyld to JITLink). |
When you say "no" visibility, do you mean local / private scope? Basically the same as JITLink's relevant concepts are:
Yep. We need to address that, but just haven't gotten to it yet. I'll try to catch up on the rest of this discussion later tonight. |
Not the same as When |
|
Thanks for the extra details @hubert-reinterpretcast.
I think so. It looks like we should map
If you have any time to elaborate on this I'd be interested to hear. It sounds like the primary motivation here is Bolt, but it'd be nice to make ORC / JITLink work for AIX JIT use cases too. |
Like ELF "internal" and "hidden" visibility, both |
I mapped both With this we should be fine to land this now @hubert-reinterpretcast @lhames. |
| return object::XCOFFSymbolRef(DRI, &Obj); | ||
| } | ||
|
|
||
| static void printSymbolEntry(raw_ostream &OS, |
There was a problem hiding this comment.
If compiling with
-DLLVM_ENABLE_ASSERTIONS=OFF
we get
../lib/ExecutionEngine/JITLink/XCOFFLinkGraphBuilder.cpp:206:13: error: unused function 'printSymbolEntry' [-Werror,-Wunused-function]
206 | static void printSymbolEntry(raw_ostream &OS,
| ^~~~~~~~~~~~~~~~
1 error generated.
There was a problem hiding this comment.
Thanks for pointing that out. I will get a fix up soon
There was a problem hiding this comment.
I have the fix up now: #134413
This patch starts the initial implementation of JITLink for XCOFF (Object format for AIX).