[XRay][RISCV] RISCV support for XRay #117368
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add RISC-V support for XRay. The RV64 implementation has been tested in both QEMU and in our production environment. Currently this requires D and C extensions, but since both RV64GC and RVA22/RVA23 are becoming mainstream, I don't think this requirement will be a big problem. Based on Ashwin Poduval's previous work: https://reviews.llvm.org/D117929 Co-authored-by: Ashwin Poduval <[email protected]>
mshockwave
requested review from
asb,
MaskRay,
sylvestre,
redstar and
deanberris
llvmbot
added
clang
|
@llvm/pr-subscribers-xray @llvm/pr-subscribers-backend-risc-v Author: Min-Yih Hsu (mshockwave) ChangesAdd RISC-V support for XRay. The RV64 implementation has been tested in both QEMU and in our production environment. Currently this requires D and C extensions, but since both RV64GC and RVA22/RVA23 are becoming mainstream, I don't think this requirement will be a big problem. Based on the previous work by @a-poduval : https://reviews.llvm.org/D117929 Patch is 33.01 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/117368.diff 14 Files Affected:
diff --git a/clang/lib/Driver/XRayArgs.cpp b/clang/lib/Driver/XRayArgs.cpp
index de5c38ebc3abbd..f8c213334a2b40 100644
--- a/clang/lib/Driver/XRayArgs.cpp
+++ b/clang/lib/Driver/XRayArgs.cpp
@@ -51,6 +51,8 @@ XRayArgs::XRayArgs(const ToolChain &TC, const ArgList &Args) {
case llvm::Triple::mips64:
case llvm::Triple::mips64el:
case llvm::Triple::systemz:
+ case llvm::Triple::riscv32:
+ case llvm::Triple::riscv64:
break;
default:
D.Diag(diag::err_drv_unsupported_opt_for_target)
diff --git a/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake b/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
index b29ae179c2b4f4..5a1e8db61023b0 100644
--- a/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
+++ b/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
@@ -102,7 +102,7 @@ if(APPLE)
set(ALL_XRAY_SUPPORTED_ARCH ${X86_64} ${ARM64})
else()
set(ALL_XRAY_SUPPORTED_ARCH ${X86_64} ${ARM32} ${ARM64} ${MIPS32} ${MIPS64}
- powerpc64le ${HEXAGON} ${LOONGARCH64})
+ powerpc64le ${HEXAGON} ${LOONGARCH64} ${RISCV32} ${RISCV64})
endif()
set(ALL_XRAY_DSO_SUPPORTED_ARCH ${X86_64} ${ARM64})
set(ALL_SHADOWCALLSTACK_SUPPORTED_ARCH ${ARM64})
diff --git a/compiler-rt/lib/xray/CMakeLists.txt b/compiler-rt/lib/xray/CMakeLists.txt
index 7e3f1a0aa616e5..e7f01a2f4f1640 100644
--- a/compiler-rt/lib/xray/CMakeLists.txt
+++ b/compiler-rt/lib/xray/CMakeLists.txt
@@ -96,6 +96,16 @@ set(hexagon_SOURCES
xray_trampoline_hexagon.S
)
+set(riscv32_SOURCES
+ xray_riscv.cpp
+ xray_trampoline_riscv32.S
+ )
+
+set(riscv64_SOURCES
+ xray_riscv.cpp
+ xray_trampoline_riscv64.S
+ )
+
set(XRAY_SOURCE_ARCHS
arm
armhf
@@ -156,6 +166,8 @@ set(XRAY_ALL_SOURCE_FILES
${mips64_SOURCES}
${mips64el_SOURCES}
${powerpc64le_SOURCES}
+ ${riscv32_SOURCES}
+ ${riscv64_SOURCES}
${XRAY_IMPL_HEADERS}
)
list(REMOVE_DUPLICATES XRAY_ALL_SOURCE_FILES)
diff --git a/compiler-rt/lib/xray/xray_interface.cpp b/compiler-rt/lib/xray/xray_interface.cpp
index b6f0e6762f1681..e66736d9a344e1 100644
--- a/compiler-rt/lib/xray/xray_interface.cpp
+++ b/compiler-rt/lib/xray/xray_interface.cpp
@@ -57,6 +57,10 @@ static const int16_t cSledLength = 64;
static const int16_t cSledLength = 8;
#elif defined(__hexagon__)
static const int16_t cSledLength = 20;
+#elif SANITIZER_RISCV64
+static const int16_t cSledLength = 76;
+#elif defined(__riscv) && (__riscv_xlen == 32)
+static const int16_t cSledLength = 60;
#else
#error "Unsupported CPU Architecture"
#endif /* CPU architecture */
diff --git a/compiler-rt/lib/xray/xray_riscv.cpp b/compiler-rt/lib/xray/xray_riscv.cpp
new file mode 100644
index 00000000000000..89ce9305ef3dbe
--- /dev/null
+++ b/compiler-rt/lib/xray/xray_riscv.cpp
@@ -0,0 +1,296 @@
+//===-- xray_riscv.cpp ----------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// Implementation of riscv-specific routines (32- and 64-bit).
+//
+//===----------------------------------------------------------------------===//
+#include "sanitizer_common/sanitizer_common.h"
+#include "xray_defs.h"
+#include "xray_interface_internal.h"
+#include <atomic>
+
+namespace __xray {
+
+// The machine codes for some instructions used in runtime patching.
+enum PatchOpcodes : uint32_t {
+ PO_ADDI = 0x00000013, // addi rd, rs1, imm
+ PO_ADD = 0x00000033, // add rd, rs1, rs2
+ PO_SW = 0x00002023, // sw rt, base(offset)
+ PO_SD = 0x00003023, // sd rt, base(offset)
+ PO_LUI = 0x00000037, // lui rd, imm
+ PO_ORI = 0x00006013, // ori rd, rs1, imm
+ PO_OR = 0x00006033, // or rd, rs1, rs2
+ PO_SLLI = 0x00001013, // slli rd, rs, shamt
+ PO_SRLI = 0x00005013, // srli rd, rs, shamt
+ PO_JALR = 0x00000067, // jalr rs
+ PO_LW = 0x00002003, // lw rd, base(offset)
+ PO_LD = 0x00003003, // ld rd, base(offset)
+ PO_J = 0x0000006f, // jal #n_bytes
+ PO_NOP = 0x00000013, // nop - pseduo-instruction, same as addi x0, x0, 0
+};
+
+enum RegNum : uint32_t {
+ RN_R0 = 0x0,
+ RN_RA = 0x1,
+ RN_SP = 0x2,
+ RN_T0 = 0x5,
+ RN_T1 = 0x6,
+ RN_T2 = 0x7,
+ RN_A0 = 0xa,
+};
+
+static inline uint32_t encodeRTypeInstruction(uint32_t Opcode, uint32_t Rs1,
+ uint32_t Rs2, uint32_t Rd) {
+ return Rs2 << 20 | Rs1 << 15 | Rd << 7 | Opcode;
+}
+
+static inline uint32_t encodeITypeInstruction(uint32_t Opcode, uint32_t Rs1,
+ uint32_t Rd, uint32_t Imm) {
+ return Imm << 20 | Rs1 << 15 | Rd << 7 | Opcode;
+}
+
+static inline uint32_t encodeSTypeInstruction(uint32_t Opcode, uint32_t Rs1,
+ uint32_t Rs2, uint32_t Imm) {
+ uint32_t imm_msbs = (Imm & 0xfe0) << 25;
+ uint32_t imm_lsbs = (Imm & 0x01f) << 7;
+ return imm_msbs | Rs2 << 20 | Rs1 << 15 | imm_lsbs | Opcode;
+}
+
+static inline uint32_t encodeUTypeInstruction(uint32_t Opcode, uint32_t Rd,
+ uint32_t Imm) {
+ return Imm << 12 | Rd << 7 | Opcode;
+}
+
+static inline uint32_t encodeJTypeInstruction(uint32_t Opcode, uint32_t Rd,
+ uint32_t Imm) {
+ uint32_t imm_msb = (Imm & 0x80000) << 31;
+ uint32_t imm_lsbs = (Imm & 0x003ff) << 21;
+ uint32_t imm_11 = (Imm & 0x00400) << 20;
+ uint32_t imm_1912 = (Imm & 0x7f800) << 12;
+ return imm_msb | imm_lsbs | imm_11 | imm_1912 | Rd << 7 | Opcode;
+}
+
+#if SANITIZER_RISCV64
+static uint32_t hi20(uint64_t val) { return (val + 0x800) >> 12; }
+static uint32_t lo12(uint64_t val) { return val & 0xfff; }
+#elif defined(__riscv) && (__riscv_xlen == 32)
+static uint32_t hi20(uint32_t val) { return (val + 0x800) >> 12; }
+static uint32_t lo12(uint32_t val) { return val & 0xfff; }
+#endif
+
+static inline bool patchSled(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled,
+ void (*TracingHook)()) XRAY_NEVER_INSTRUMENT {
+ // When |Enable| == true,
+ // We replace the following compile-time stub (sled):
+ //
+ // xray_sled_n:
+ // J .tmpN
+ // 29 or 37 C.NOPs (58 or 74 bytes)
+ // .tmpN
+ //
+ // With one of the following runtime patches:
+ //
+ // xray_sled_n (32-bit):
+ // addi sp, sp, -16 ;create stack frame
+ // sw ra, 12(sp) ;save return address
+ // sw t2, 8(sp) ;save register t2
+ // sw t1, 4(sp) ;save register t1
+ // sw a0, 0(sp) ;save register a0
+ // lui t1, %hi(__xray_FunctionEntry/Exit)
+ // addi t1, t1, %lo(__xray_FunctionEntry/Exit)
+ // lui a0, %hi(function_id)
+ // addi a0, a0, %lo(function_id) ;pass function id
+ // jalr t1 ;call Tracing hook
+ // lw a0, 0(sp) ;restore register a0
+ // lw t1, 4(sp) ;restore register t1
+ // lw t2, 8(sp) ;restore register t2
+ // lw ra, 12(sp) ;restore return address
+ // addi sp, sp, 16 ;delete stack frame
+ //
+ // xray_sled_n (64-bit):
+ // addi sp, sp, -32 ;create stack frame
+ // sd ra, 24(sp) ;save return address
+ // sd t2, 16(sp) ;save register t2
+ // sd t1, 8(sp) ;save register t1
+ // sd a0, 0(sp) ;save register a0
+ // lui t2, %highest(__xray_FunctionEntry/Exit)
+ // addi t2, t2, %higher(__xray_FunctionEntry/Exit)
+ // slli t2, t2, 32
+ // lui t1, t1, %hi(__xray_FunctionEntry/Exit)
+ // addi t1, t1, %lo(__xray_FunctionEntry/Exit)
+ // add t1, t2, t1
+ // lui a0, %hi(function_id)
+ // addi a0, a0, %lo(function_id) ;pass function id
+ // jalr t1 ;call Tracing hook
+ // ld a0, 0(sp) ;restore register a0
+ // ld t1, 8(sp) ;restore register t1
+ // ld t2, 16(sp) ;restore register t2
+ // ld ra, 24(sp) ;restore return address
+ // addi sp, sp, 32 ;delete stack frame
+ //
+ // Replacement of the first 4-byte instruction should be the last and atomic
+ // operation, so that the user code which reaches the sled concurrently
+ // either jumps over the whole sled, or executes the whole sled when the
+ // latter is ready.
+ //
+ // When |Enable|==false, we set back the first instruction in the sled to be
+ // J 60 bytes (rv32)
+ // J 76 bytes (rv64)
+
+ uint32_t *Address = reinterpret_cast<uint32_t *>(Sled.address());
+ if (Enable) {
+ // If the ISA is RISCV 64, the Tracing Hook needs to be typecast to a 64 bit
+ // value
+#if SANITIZER_RISCV64
+ uint32_t LoTracingHookAddr = lo12(reinterpret_cast<uint64_t>(TracingHook));
+ uint32_t HiTracingHookAddr = hi20(reinterpret_cast<uint64_t>(TracingHook));
+ uint32_t HigherTracingHookAddr =
+ lo12((reinterpret_cast<uint64_t>(TracingHook) + 0x80000000) >> 32);
+ uint32_t HighestTracingHookAddr =
+ hi20((reinterpret_cast<uint64_t>(TracingHook) + 0x80000000) >> 32);
+ // We typecast the Tracing Hook to a 32 bit value for RISCV32
+#elif defined(__riscv) && (__riscv_xlen == 32)
+ uint32_t LoTracingHookAddr = lo12(reinterpret_cast<uint32_t>(TracingHook));
+ uint32_t HiTracingHookAddr = hi20((reinterpret_cast<uint32_t>(TracingHook));
+#endif
+ uint32_t LoFunctionID = lo12(FuncId);
+ uint32_t HiFunctionID = hi20(FuncId);
+ // The sled that is patched in for RISCV64 defined below. We need the entire
+ // sleds corresponding to both ISAs to be protected by defines because the
+ // first few instructions are all different, because we store doubles in
+ // case of RV64 and store words for RV32. Subsequently, we have LUI - and in
+ // case of RV64, we need extra instructions from this point on, so we see
+ // differences in addresses to which instructions are stored.
+#if SANITIZER_RISCV64
+ Address[1] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_RA, 0x18);
+ Address[2] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_T2, 0x10);
+ Address[3] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_T1, 0x8);
+ Address[4] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[5] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_T2,
+ HighestTracingHookAddr);
+ Address[6] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_T2,
+ RegNum::RN_T2, HigherTracingHookAddr);
+ Address[7] = encodeITypeInstruction(PatchOpcodes::PO_SLLI, RegNum::RN_T2,
+ RegNum::RN_T2, 0x20);
+ Address[8] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_T1,
+ HiTracingHookAddr);
+ Address[9] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_T1,
+ RegNum::RN_T1, LoTracingHookAddr);
+ Address[10] = encodeRTypeInstruction(PatchOpcodes::PO_ADD, RegNum::RN_T1,
+ RegNum::RN_T2, RegNum::RN_T1);
+ Address[11] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_A0,
+ HiFunctionID);
+ Address[12] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_A0,
+ RegNum::RN_A0, LoFunctionID);
+ Address[13] = encodeITypeInstruction(PatchOpcodes::PO_JALR, RegNum::RN_T1,
+ RegNum::RN_RA, 0x0);
+ Address[14] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[15] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_T1, 0x8);
+ Address[16] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_T2, 0x10);
+ Address[17] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_RA, 0x18);
+ Address[18] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_SP,
+ RegNum::RN_SP, 0x20);
+ uint32_t CreateStackSpace = encodeITypeInstruction(
+ PatchOpcodes::PO_ADDI, RegNum::RN_SP, RegNum::RN_SP, 0xffe0);
+#elif defined(__riscv) && (__riscv_xlen == 32)
+ Address[1] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_RA, 0x0c);
+ Address[2] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_T2, 0x08);
+ Address[3] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_T1, 0x4);
+ Address[4] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[5] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_T1,
+ HiTracingHookAddr);
+ Address[6] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_T1,
+ RegNum::RN_T1, LoTracingHookAddr);
+ Address[7] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_A0,
+ HiFunctionID);
+ Address[8] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_A0,
+ RegNum::RN_A0, LoFunctionID);
+ Address[9] = encodeITypeInstruction(PatchOpcodes::PO_JALR, RegNum::RN_T1,
+ RegNum::RN_RA, 0x0);
+ Address[10] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[11] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_T1, 0x4);
+ Address[12] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_T2, 0x08);
+ Address[13] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_RA, 0x0c);
+ Address[14] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_SP,
+ RegNum::RN_SP, 0x10);
+ uint32_t CreateStackSpace = encodeITypeInstruction(
+ PatchOpcodes::PO_ADDI, RegNum::RN_SP, RegNum::RN_SP, 0xfff0);
+#endif
+ std::atomic_store_explicit(
+ reinterpret_cast<std::atomic<uint32_t> *>(Address), CreateStackSpace,
+ std::memory_order_release);
+ } else {
+ uint32_t CreateBranch = encodeJTypeInstruction(
+ // Jump distance is different in both ISAs due to difference in size of
+ // sleds
+#if SANITIZER_RISCV64
+ PatchOpcodes::PO_J, RegNum::RN_R0,
+ 0x026); // jump encodes an offset in multiples of 2 bytes. 38*2 = 76
+#elif defined(__riscv) && (__riscv_xlen == 32)
+ PatchOpcodes::PO_J, RegNum::RN_R0,
+ 0x01e); // jump encodes an offset in multiples of 2 bytes. 30*2 = 60
+#endif
+ std::atomic_store_explicit(
+ reinterpret_cast<std::atomic<uint32_t> *>(Address), CreateBranch,
+ std::memory_order_release);
+ }
+ return true;
+}
+
+bool patchFunctionEntry(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled,
+ const XRayTrampolines &Trampolines,
+ bool LogArgs) XRAY_NEVER_INSTRUMENT {
+ // We don't support Logging argument at this moment, so we always
+ // use EntryTrampoline.
+ return patchSled(Enable, FuncId, Sled, Trampolines.EntryTrampoline);
+}
+
+bool patchFunctionExit(
+ const bool Enable, const uint32_t FuncId, const XRaySledEntry &Sled,
+ const XRayTrampolines &Trampolines) XRAY_NEVER_INSTRUMENT {
+ return patchSled(Enable, FuncId, Sled, Trampolines.ExitTrampoline);
+}
+
+bool patchFunctionTailExit(
+ const bool Enable, const uint32_t FuncId, const XRaySledEntry &Sled,
+ const XRayTrampolines &Trampolines) XRAY_NEVER_INSTRUMENT {
+ return patchSled(Enable, FuncId, Sled, Trampolines.TailExitTrampoline);
+}
+
+bool patchCustomEvent(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+ return false;
+}
+
+bool patchTypedEvent(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+ return false;
+}
+} // namespace __xray
+
+extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT {}
diff --git a/compiler-rt/lib/xray/xray_trampoline_riscv32.S b/compiler-rt/lib/xray/xray_trampoline_riscv32.S
new file mode 100644
index 00000000000000..9916e0321d24fd
--- /dev/null
+++ b/compiler-rt/lib/xray/xray_trampoline_riscv32.S
@@ -0,0 +1,83 @@
+//===-- xray_trampoline_riscv32.s ----------------------------------*- ASM -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// This implements the riscv32-specific assembler for the trampolines.
+//
+//===----------------------------------------------------------------------===//
+
+.macro SAVE_ARG_REGISTERS
+ // Push argument registers to stack
+ addi sp, sp, -100
+ .cfi_def_cfa_offset 100
+ sw ra, 96(sp)
+ .cfi_offset ra, -4
+ sw a7, 92(sp)
+ sw a6, 88(sp)
+ sw a5, 84(sp)
+ sw a4, 80(sp)
+ sw a3, 76(sp)
+ sw a2, 72(sp)
+ sw a1, 68(sp)
+ sw a0, 64(sp)
+ fsd fa7, 56(sp)
+ fsd fa6, 48(sp)
+ fsd fa5, 40(sp)
+ fsd fa4, 32(sp)
+ fsd fa3, 24(sp)
+ fsd fa2, 16(sp)
+ fsd fa1, 8(sp)
+ fsd fa0, 0(sp)
+.endm
+
+.macro RESTORE_ARG_REGISTERS
+ // Restore argument registers
+ fld fa0, 0(sp)
+ fld fa1, 8(sp)
+ fld fa2, 16(sp)
+ fld fa3, 24(sp)
+ fld fa4, 32(sp)
+ fld fa5, 40(sp)
+ fld fa6, 48(sp)
+ fld fa7, 56(sp)
+ lw a0, 64(sp)
+ lw a1, 68(sp)
+ lw a2, 72(sp)
+ lw a3, 76(sp)
+ lw a4, 80(sp)
+ lw a5, 84(sp)
+ lw a6, 88(sp)
+ lw a7, 92(sp)
+ lw ra, 96(sp)
+ addi sp, sp, 100
+.endm
+
+.macro SAVE_RET_REGISTERS
+ // Push return registers to stack
+ addi sp, sp, -28
+ .cfi_def_cfa_offset 28
+ sw ra, 24(sp)
+ .cfi_offset ra, -4
+ sw a1, 20(sp)
+ sw a0, 16(sp)
+ fsd fa1, 8(sp)
+ fsd fa0, 0(sp)
+.endm
+
+.macro RESTORE_RET_REGISTERS
+ // Restore return registers
+ fld fa0, 0(sp)
+ fld fa1, 8(sp)
+ lw a0, 16(sp)
+ lw a1, 20(sp)
+ lw ra, 24(sp)
+ addi sp, sp, 28
+.endm
+
+#include "xray_trampoline_riscv_common.S"
diff --git a/compiler-rt/lib/xray/xray_trampoline_riscv64.S b/compiler-rt/lib/xray/xray_trampoline_riscv64.S
new file mode 100644
index 00000000000000..102b9881567d9a
--- /dev/null
+++ b/compiler-rt/lib/xray/xray_trampoline_riscv64.S
@@ -0,0 +1,83 @@
+//===-- xray_trampoline_riscv64.s ----------------------------------*- ASM -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This fil...
[truncated]
|
|
@llvm/pr-subscribers-clang-driver Author: Min-Yih Hsu (mshockwave) ChangesAdd RISC-V support for XRay. The RV64 implementation has been tested in both QEMU and in our production environment. Currently this requires D and C extensions, but since both RV64GC and RVA22/RVA23 are becoming mainstream, I don't think this requirement will be a big problem. Based on the previous work by @a-poduval : https://reviews.llvm.org/D117929 Patch is 33.01 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/117368.diff 14 Files Affected:
diff --git a/clang/lib/Driver/XRayArgs.cpp b/clang/lib/Driver/XRayArgs.cpp
index de5c38ebc3abbd..f8c213334a2b40 100644
--- a/clang/lib/Driver/XRayArgs.cpp
+++ b/clang/lib/Driver/XRayArgs.cpp
@@ -51,6 +51,8 @@ XRayArgs::XRayArgs(const ToolChain &TC, const ArgList &Args) {
case llvm::Triple::mips64:
case llvm::Triple::mips64el:
case llvm::Triple::systemz:
+ case llvm::Triple::riscv32:
+ case llvm::Triple::riscv64:
break;
default:
D.Diag(diag::err_drv_unsupported_opt_for_target)
diff --git a/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake b/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
index b29ae179c2b4f4..5a1e8db61023b0 100644
--- a/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
+++ b/compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake
@@ -102,7 +102,7 @@ if(APPLE)
set(ALL_XRAY_SUPPORTED_ARCH ${X86_64} ${ARM64})
else()
set(ALL_XRAY_SUPPORTED_ARCH ${X86_64} ${ARM32} ${ARM64} ${MIPS32} ${MIPS64}
- powerpc64le ${HEXAGON} ${LOONGARCH64})
+ powerpc64le ${HEXAGON} ${LOONGARCH64} ${RISCV32} ${RISCV64})
endif()
set(ALL_XRAY_DSO_SUPPORTED_ARCH ${X86_64} ${ARM64})
set(ALL_SHADOWCALLSTACK_SUPPORTED_ARCH ${ARM64})
diff --git a/compiler-rt/lib/xray/CMakeLists.txt b/compiler-rt/lib/xray/CMakeLists.txt
index 7e3f1a0aa616e5..e7f01a2f4f1640 100644
--- a/compiler-rt/lib/xray/CMakeLists.txt
+++ b/compiler-rt/lib/xray/CMakeLists.txt
@@ -96,6 +96,16 @@ set(hexagon_SOURCES
xray_trampoline_hexagon.S
)
+set(riscv32_SOURCES
+ xray_riscv.cpp
+ xray_trampoline_riscv32.S
+ )
+
+set(riscv64_SOURCES
+ xray_riscv.cpp
+ xray_trampoline_riscv64.S
+ )
+
set(XRAY_SOURCE_ARCHS
arm
armhf
@@ -156,6 +166,8 @@ set(XRAY_ALL_SOURCE_FILES
${mips64_SOURCES}
${mips64el_SOURCES}
${powerpc64le_SOURCES}
+ ${riscv32_SOURCES}
+ ${riscv64_SOURCES}
${XRAY_IMPL_HEADERS}
)
list(REMOVE_DUPLICATES XRAY_ALL_SOURCE_FILES)
diff --git a/compiler-rt/lib/xray/xray_interface.cpp b/compiler-rt/lib/xray/xray_interface.cpp
index b6f0e6762f1681..e66736d9a344e1 100644
--- a/compiler-rt/lib/xray/xray_interface.cpp
+++ b/compiler-rt/lib/xray/xray_interface.cpp
@@ -57,6 +57,10 @@ static const int16_t cSledLength = 64;
static const int16_t cSledLength = 8;
#elif defined(__hexagon__)
static const int16_t cSledLength = 20;
+#elif SANITIZER_RISCV64
+static const int16_t cSledLength = 76;
+#elif defined(__riscv) && (__riscv_xlen == 32)
+static const int16_t cSledLength = 60;
#else
#error "Unsupported CPU Architecture"
#endif /* CPU architecture */
diff --git a/compiler-rt/lib/xray/xray_riscv.cpp b/compiler-rt/lib/xray/xray_riscv.cpp
new file mode 100644
index 00000000000000..89ce9305ef3dbe
--- /dev/null
+++ b/compiler-rt/lib/xray/xray_riscv.cpp
@@ -0,0 +1,296 @@
+//===-- xray_riscv.cpp ----------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// Implementation of riscv-specific routines (32- and 64-bit).
+//
+//===----------------------------------------------------------------------===//
+#include "sanitizer_common/sanitizer_common.h"
+#include "xray_defs.h"
+#include "xray_interface_internal.h"
+#include <atomic>
+
+namespace __xray {
+
+// The machine codes for some instructions used in runtime patching.
+enum PatchOpcodes : uint32_t {
+ PO_ADDI = 0x00000013, // addi rd, rs1, imm
+ PO_ADD = 0x00000033, // add rd, rs1, rs2
+ PO_SW = 0x00002023, // sw rt, base(offset)
+ PO_SD = 0x00003023, // sd rt, base(offset)
+ PO_LUI = 0x00000037, // lui rd, imm
+ PO_ORI = 0x00006013, // ori rd, rs1, imm
+ PO_OR = 0x00006033, // or rd, rs1, rs2
+ PO_SLLI = 0x00001013, // slli rd, rs, shamt
+ PO_SRLI = 0x00005013, // srli rd, rs, shamt
+ PO_JALR = 0x00000067, // jalr rs
+ PO_LW = 0x00002003, // lw rd, base(offset)
+ PO_LD = 0x00003003, // ld rd, base(offset)
+ PO_J = 0x0000006f, // jal #n_bytes
+ PO_NOP = 0x00000013, // nop - pseduo-instruction, same as addi x0, x0, 0
+};
+
+enum RegNum : uint32_t {
+ RN_R0 = 0x0,
+ RN_RA = 0x1,
+ RN_SP = 0x2,
+ RN_T0 = 0x5,
+ RN_T1 = 0x6,
+ RN_T2 = 0x7,
+ RN_A0 = 0xa,
+};
+
+static inline uint32_t encodeRTypeInstruction(uint32_t Opcode, uint32_t Rs1,
+ uint32_t Rs2, uint32_t Rd) {
+ return Rs2 << 20 | Rs1 << 15 | Rd << 7 | Opcode;
+}
+
+static inline uint32_t encodeITypeInstruction(uint32_t Opcode, uint32_t Rs1,
+ uint32_t Rd, uint32_t Imm) {
+ return Imm << 20 | Rs1 << 15 | Rd << 7 | Opcode;
+}
+
+static inline uint32_t encodeSTypeInstruction(uint32_t Opcode, uint32_t Rs1,
+ uint32_t Rs2, uint32_t Imm) {
+ uint32_t imm_msbs = (Imm & 0xfe0) << 25;
+ uint32_t imm_lsbs = (Imm & 0x01f) << 7;
+ return imm_msbs | Rs2 << 20 | Rs1 << 15 | imm_lsbs | Opcode;
+}
+
+static inline uint32_t encodeUTypeInstruction(uint32_t Opcode, uint32_t Rd,
+ uint32_t Imm) {
+ return Imm << 12 | Rd << 7 | Opcode;
+}
+
+static inline uint32_t encodeJTypeInstruction(uint32_t Opcode, uint32_t Rd,
+ uint32_t Imm) {
+ uint32_t imm_msb = (Imm & 0x80000) << 31;
+ uint32_t imm_lsbs = (Imm & 0x003ff) << 21;
+ uint32_t imm_11 = (Imm & 0x00400) << 20;
+ uint32_t imm_1912 = (Imm & 0x7f800) << 12;
+ return imm_msb | imm_lsbs | imm_11 | imm_1912 | Rd << 7 | Opcode;
+}
+
+#if SANITIZER_RISCV64
+static uint32_t hi20(uint64_t val) { return (val + 0x800) >> 12; }
+static uint32_t lo12(uint64_t val) { return val & 0xfff; }
+#elif defined(__riscv) && (__riscv_xlen == 32)
+static uint32_t hi20(uint32_t val) { return (val + 0x800) >> 12; }
+static uint32_t lo12(uint32_t val) { return val & 0xfff; }
+#endif
+
+static inline bool patchSled(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled,
+ void (*TracingHook)()) XRAY_NEVER_INSTRUMENT {
+ // When |Enable| == true,
+ // We replace the following compile-time stub (sled):
+ //
+ // xray_sled_n:
+ // J .tmpN
+ // 29 or 37 C.NOPs (58 or 74 bytes)
+ // .tmpN
+ //
+ // With one of the following runtime patches:
+ //
+ // xray_sled_n (32-bit):
+ // addi sp, sp, -16 ;create stack frame
+ // sw ra, 12(sp) ;save return address
+ // sw t2, 8(sp) ;save register t2
+ // sw t1, 4(sp) ;save register t1
+ // sw a0, 0(sp) ;save register a0
+ // lui t1, %hi(__xray_FunctionEntry/Exit)
+ // addi t1, t1, %lo(__xray_FunctionEntry/Exit)
+ // lui a0, %hi(function_id)
+ // addi a0, a0, %lo(function_id) ;pass function id
+ // jalr t1 ;call Tracing hook
+ // lw a0, 0(sp) ;restore register a0
+ // lw t1, 4(sp) ;restore register t1
+ // lw t2, 8(sp) ;restore register t2
+ // lw ra, 12(sp) ;restore return address
+ // addi sp, sp, 16 ;delete stack frame
+ //
+ // xray_sled_n (64-bit):
+ // addi sp, sp, -32 ;create stack frame
+ // sd ra, 24(sp) ;save return address
+ // sd t2, 16(sp) ;save register t2
+ // sd t1, 8(sp) ;save register t1
+ // sd a0, 0(sp) ;save register a0
+ // lui t2, %highest(__xray_FunctionEntry/Exit)
+ // addi t2, t2, %higher(__xray_FunctionEntry/Exit)
+ // slli t2, t2, 32
+ // lui t1, t1, %hi(__xray_FunctionEntry/Exit)
+ // addi t1, t1, %lo(__xray_FunctionEntry/Exit)
+ // add t1, t2, t1
+ // lui a0, %hi(function_id)
+ // addi a0, a0, %lo(function_id) ;pass function id
+ // jalr t1 ;call Tracing hook
+ // ld a0, 0(sp) ;restore register a0
+ // ld t1, 8(sp) ;restore register t1
+ // ld t2, 16(sp) ;restore register t2
+ // ld ra, 24(sp) ;restore return address
+ // addi sp, sp, 32 ;delete stack frame
+ //
+ // Replacement of the first 4-byte instruction should be the last and atomic
+ // operation, so that the user code which reaches the sled concurrently
+ // either jumps over the whole sled, or executes the whole sled when the
+ // latter is ready.
+ //
+ // When |Enable|==false, we set back the first instruction in the sled to be
+ // J 60 bytes (rv32)
+ // J 76 bytes (rv64)
+
+ uint32_t *Address = reinterpret_cast<uint32_t *>(Sled.address());
+ if (Enable) {
+ // If the ISA is RISCV 64, the Tracing Hook needs to be typecast to a 64 bit
+ // value
+#if SANITIZER_RISCV64
+ uint32_t LoTracingHookAddr = lo12(reinterpret_cast<uint64_t>(TracingHook));
+ uint32_t HiTracingHookAddr = hi20(reinterpret_cast<uint64_t>(TracingHook));
+ uint32_t HigherTracingHookAddr =
+ lo12((reinterpret_cast<uint64_t>(TracingHook) + 0x80000000) >> 32);
+ uint32_t HighestTracingHookAddr =
+ hi20((reinterpret_cast<uint64_t>(TracingHook) + 0x80000000) >> 32);
+ // We typecast the Tracing Hook to a 32 bit value for RISCV32
+#elif defined(__riscv) && (__riscv_xlen == 32)
+ uint32_t LoTracingHookAddr = lo12(reinterpret_cast<uint32_t>(TracingHook));
+ uint32_t HiTracingHookAddr = hi20((reinterpret_cast<uint32_t>(TracingHook));
+#endif
+ uint32_t LoFunctionID = lo12(FuncId);
+ uint32_t HiFunctionID = hi20(FuncId);
+ // The sled that is patched in for RISCV64 defined below. We need the entire
+ // sleds corresponding to both ISAs to be protected by defines because the
+ // first few instructions are all different, because we store doubles in
+ // case of RV64 and store words for RV32. Subsequently, we have LUI - and in
+ // case of RV64, we need extra instructions from this point on, so we see
+ // differences in addresses to which instructions are stored.
+#if SANITIZER_RISCV64
+ Address[1] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_RA, 0x18);
+ Address[2] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_T2, 0x10);
+ Address[3] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_T1, 0x8);
+ Address[4] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[5] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_T2,
+ HighestTracingHookAddr);
+ Address[6] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_T2,
+ RegNum::RN_T2, HigherTracingHookAddr);
+ Address[7] = encodeITypeInstruction(PatchOpcodes::PO_SLLI, RegNum::RN_T2,
+ RegNum::RN_T2, 0x20);
+ Address[8] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_T1,
+ HiTracingHookAddr);
+ Address[9] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_T1,
+ RegNum::RN_T1, LoTracingHookAddr);
+ Address[10] = encodeRTypeInstruction(PatchOpcodes::PO_ADD, RegNum::RN_T1,
+ RegNum::RN_T2, RegNum::RN_T1);
+ Address[11] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_A0,
+ HiFunctionID);
+ Address[12] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_A0,
+ RegNum::RN_A0, LoFunctionID);
+ Address[13] = encodeITypeInstruction(PatchOpcodes::PO_JALR, RegNum::RN_T1,
+ RegNum::RN_RA, 0x0);
+ Address[14] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[15] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_T1, 0x8);
+ Address[16] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_T2, 0x10);
+ Address[17] = encodeITypeInstruction(PatchOpcodes::PO_LD, RegNum::RN_SP,
+ RegNum::RN_RA, 0x18);
+ Address[18] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_SP,
+ RegNum::RN_SP, 0x20);
+ uint32_t CreateStackSpace = encodeITypeInstruction(
+ PatchOpcodes::PO_ADDI, RegNum::RN_SP, RegNum::RN_SP, 0xffe0);
+#elif defined(__riscv) && (__riscv_xlen == 32)
+ Address[1] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_RA, 0x0c);
+ Address[2] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_T2, 0x08);
+ Address[3] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_T1, 0x4);
+ Address[4] = encodeSTypeInstruction(PatchOpcodes::PO_SW, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[5] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_T1,
+ HiTracingHookAddr);
+ Address[6] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_T1,
+ RegNum::RN_T1, LoTracingHookAddr);
+ Address[7] = encodeUTypeInstruction(PatchOpcodes::PO_LUI, RegNum::RN_A0,
+ HiFunctionID);
+ Address[8] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_A0,
+ RegNum::RN_A0, LoFunctionID);
+ Address[9] = encodeITypeInstruction(PatchOpcodes::PO_JALR, RegNum::RN_T1,
+ RegNum::RN_RA, 0x0);
+ Address[10] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_A0, 0x0);
+ Address[11] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_T1, 0x4);
+ Address[12] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_T2, 0x08);
+ Address[13] = encodeITypeInstruction(PatchOpcodes::PO_LW, RegNum::RN_SP,
+ RegNum::RN_RA, 0x0c);
+ Address[14] = encodeITypeInstruction(PatchOpcodes::PO_ADDI, RegNum::RN_SP,
+ RegNum::RN_SP, 0x10);
+ uint32_t CreateStackSpace = encodeITypeInstruction(
+ PatchOpcodes::PO_ADDI, RegNum::RN_SP, RegNum::RN_SP, 0xfff0);
+#endif
+ std::atomic_store_explicit(
+ reinterpret_cast<std::atomic<uint32_t> *>(Address), CreateStackSpace,
+ std::memory_order_release);
+ } else {
+ uint32_t CreateBranch = encodeJTypeInstruction(
+ // Jump distance is different in both ISAs due to difference in size of
+ // sleds
+#if SANITIZER_RISCV64
+ PatchOpcodes::PO_J, RegNum::RN_R0,
+ 0x026); // jump encodes an offset in multiples of 2 bytes. 38*2 = 76
+#elif defined(__riscv) && (__riscv_xlen == 32)
+ PatchOpcodes::PO_J, RegNum::RN_R0,
+ 0x01e); // jump encodes an offset in multiples of 2 bytes. 30*2 = 60
+#endif
+ std::atomic_store_explicit(
+ reinterpret_cast<std::atomic<uint32_t> *>(Address), CreateBranch,
+ std::memory_order_release);
+ }
+ return true;
+}
+
+bool patchFunctionEntry(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled,
+ const XRayTrampolines &Trampolines,
+ bool LogArgs) XRAY_NEVER_INSTRUMENT {
+ // We don't support Logging argument at this moment, so we always
+ // use EntryTrampoline.
+ return patchSled(Enable, FuncId, Sled, Trampolines.EntryTrampoline);
+}
+
+bool patchFunctionExit(
+ const bool Enable, const uint32_t FuncId, const XRaySledEntry &Sled,
+ const XRayTrampolines &Trampolines) XRAY_NEVER_INSTRUMENT {
+ return patchSled(Enable, FuncId, Sled, Trampolines.ExitTrampoline);
+}
+
+bool patchFunctionTailExit(
+ const bool Enable, const uint32_t FuncId, const XRaySledEntry &Sled,
+ const XRayTrampolines &Trampolines) XRAY_NEVER_INSTRUMENT {
+ return patchSled(Enable, FuncId, Sled, Trampolines.TailExitTrampoline);
+}
+
+bool patchCustomEvent(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+ return false;
+}
+
+bool patchTypedEvent(const bool Enable, const uint32_t FuncId,
+ const XRaySledEntry &Sled) XRAY_NEVER_INSTRUMENT {
+ return false;
+}
+} // namespace __xray
+
+extern "C" void __xray_ArgLoggerEntry() XRAY_NEVER_INSTRUMENT {}
diff --git a/compiler-rt/lib/xray/xray_trampoline_riscv32.S b/compiler-rt/lib/xray/xray_trampoline_riscv32.S
new file mode 100644
index 00000000000000..9916e0321d24fd
--- /dev/null
+++ b/compiler-rt/lib/xray/xray_trampoline_riscv32.S
@@ -0,0 +1,83 @@
+//===-- xray_trampoline_riscv32.s ----------------------------------*- ASM -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of XRay, a dynamic runtime instrumentation system.
+//
+// This implements the riscv32-specific assembler for the trampolines.
+//
+//===----------------------------------------------------------------------===//
+
+.macro SAVE_ARG_REGISTERS
+ // Push argument registers to stack
+ addi sp, sp, -100
+ .cfi_def_cfa_offset 100
+ sw ra, 96(sp)
+ .cfi_offset ra, -4
+ sw a7, 92(sp)
+ sw a6, 88(sp)
+ sw a5, 84(sp)
+ sw a4, 80(sp)
+ sw a3, 76(sp)
+ sw a2, 72(sp)
+ sw a1, 68(sp)
+ sw a0, 64(sp)
+ fsd fa7, 56(sp)
+ fsd fa6, 48(sp)
+ fsd fa5, 40(sp)
+ fsd fa4, 32(sp)
+ fsd fa3, 24(sp)
+ fsd fa2, 16(sp)
+ fsd fa1, 8(sp)
+ fsd fa0, 0(sp)
+.endm
+
+.macro RESTORE_ARG_REGISTERS
+ // Restore argument registers
+ fld fa0, 0(sp)
+ fld fa1, 8(sp)
+ fld fa2, 16(sp)
+ fld fa3, 24(sp)
+ fld fa4, 32(sp)
+ fld fa5, 40(sp)
+ fld fa6, 48(sp)
+ fld fa7, 56(sp)
+ lw a0, 64(sp)
+ lw a1, 68(sp)
+ lw a2, 72(sp)
+ lw a3, 76(sp)
+ lw a4, 80(sp)
+ lw a5, 84(sp)
+ lw a6, 88(sp)
+ lw a7, 92(sp)
+ lw ra, 96(sp)
+ addi sp, sp, 100
+.endm
+
+.macro SAVE_RET_REGISTERS
+ // Push return registers to stack
+ addi sp, sp, -28
+ .cfi_def_cfa_offset 28
+ sw ra, 24(sp)
+ .cfi_offset ra, -4
+ sw a1, 20(sp)
+ sw a0, 16(sp)
+ fsd fa1, 8(sp)
+ fsd fa0, 0(sp)
+.endm
+
+.macro RESTORE_RET_REGISTERS
+ // Restore return registers
+ fld fa0, 0(sp)
+ fld fa1, 8(sp)
+ lw a0, 16(sp)
+ lw a1, 20(sp)
+ lw ra, 24(sp)
+ addi sp, sp, 28
+.endm
+
+#include "xray_trampoline_riscv_common.S"
diff --git a/compiler-rt/lib/xray/xray_trampoline_riscv64.S b/compiler-rt/lib/xray/xray_trampoline_riscv64.S
new file mode 100644
index 00000000000000..102b9881567d9a
--- /dev/null
+++ b/compiler-rt/lib/xray/xray_trampoline_riscv64.S
@@ -0,0 +1,83 @@
+//===-- xray_trampoline_riscv64.s ----------------------------------*- ASM -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This fil...
[truncated]
|
jrtc27
reviewed
Nov 22, 2024
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| enum PatchOpcodes : uint32_t { | ||
| PO_ADDI = 0x00000013, // addi rd, rs1, imm | ||
| PO_ADD = 0x00000033, // add rd, rs1, rs2 | ||
| PO_SW = 0x00002023, // sw rt, base(offset) |
There was a problem hiding this comment.
rs2, imm(rs1); names don't match the architecture (rt looks MIPSy) and the order of base and offset is the wrong way round
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| PO_LUI = 0x00000037, // lui rd, imm | ||
| PO_ORI = 0x00006013, // ori rd, rs1, imm | ||
| PO_OR = 0x00006033, // or rd, rs1, rs2 | ||
| PO_SLLI = 0x00001013, // slli rd, rs, shamt |
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| PO_OR = 0x00006033, // or rd, rs1, rs2 | ||
| PO_SLLI = 0x00001013, // slli rd, rs, shamt | ||
| PO_SRLI = 0x00005013, // srli rd, rs, shamt | ||
| PO_JALR = 0x00000067, // jalr rs |
There was a problem hiding this comment.
rd, rs1? No rs, and you've not encoded rd=ra here AFAICT
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| PO_JALR = 0x00000067, // jalr rs | ||
| PO_LW = 0x00002003, // lw rd, base(offset) | ||
| PO_LD = 0x00003003, // ld rd, base(offset) | ||
| PO_J = 0x0000006f, // jal #n_bytes |
There was a problem hiding this comment.
That's a bit misleading; JAL uses imm[0] to instead be imm[20], but this makes it look like the encoding puts the number of bytes in rather than the number of 16-bit parcels. I'd just go with jal imm.
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| PO_LW = 0x00002003, // lw rd, base(offset) | ||
| PO_LD = 0x00003003, // ld rd, base(offset) | ||
| PO_J = 0x0000006f, // jal #n_bytes | ||
| PO_NOP = 0x00000013, // nop - pseduo-instruction, same as addi x0, x0, 0 |
There was a problem hiding this comment.
"pseudo", though I don't think you need to explain what a nop is, the only thing worth noting is what it's a (canonical) alias for.
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| }; | ||
|
|
||
| enum RegNum : uint32_t { | ||
| RN_R0 = 0x0, |
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| RN_T0 = 0x5, | ||
| RN_T1 = 0x6, | ||
| RN_T2 = 0x7, | ||
| RN_A0 = 0xa, |
There was a problem hiding this comment.
Does hex make these more readable? I feel decimal would be better, given people tend to know them by their xN equivalents, written in decimal.
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| } | ||
|
|
||
| #if SANITIZER_RISCV64 | ||
| static uint32_t hi20(uint64_t val) { return (val + 0x800) >> 12; } |
There was a problem hiding this comment.
uint64_t input is odd, you can't encode the upper bits. Plus this doesn't mask, so if any of them are set the low 12 upper bits will end up in the return value. As it happens this doesn't matter since the only uses give it to encodeUTypeInstruction, which will shift back up and truncate, but it's still pretty dodgy code. uint32_t wouldn't have that issue.
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| // differences in addresses to which instructions are stored. | ||
| #if SANITIZER_RISCV64 | ||
| Address[1] = encodeSTypeInstruction(PatchOpcodes::PO_SD, RegNum::RN_SP, | ||
| RegNum::RN_RA, 0x18); |
There was a problem hiding this comment.
Can we avoid all these hard-coded magic immediates? I get they're documented in the assembly above, but surely they can be written in terms of other things, i.e. frameindex * xlen_bytes?
There was a problem hiding this comment.
Then you should also be able to share most of the code between 32-bit and 64-bit, with just a few opcode variables (e.g. LW/SW vs LD/SD)
There was a problem hiding this comment.
Good idea, it's fixed now.
jrtc27
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| 0x026); // jump encodes an offset in multiples of 2 bytes. 38*2 = 76 | ||
| #elif defined(__riscv) && (__riscv_xlen == 32) | ||
| PatchOpcodes::PO_J, RegNum::RN_R0, | ||
| 0x01e); // jump encodes an offset in multiples of 2 bytes. 30*2 = 60 |
There was a problem hiding this comment.
Use cSledLength rather than separate magic constants? Also, not sure if I'd instead expect encodeJTypeInstruction to take in the value with the LSB present as 0 and then strip it off. Matches more the assembly (contrast with lui/auipc where the assembly doesn't include the 12 LSBs).
There was a problem hiding this comment.
Use cSledLength rather than separate magic constants?
I don't think I can access cSledLength here, as it's a static variable in another CPP file.
if I'd instead expect encodeJTypeInstruction to take in the value with the LSB present as 0 and then strip it off.
It's fixed now.
topperc
reviewed
Nov 22, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| // | ||
| // This file is a part of XRay, a dynamic runtime instrumentation system. | ||
| // | ||
| // Implementation of riscv-specific routines (32- and 64-bit). |
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
|
|
||
| static inline uint32_t encodeSTypeInstruction(uint32_t Opcode, uint32_t Rs1, | ||
| uint32_t Rs2, uint32_t Imm) { | ||
| uint32_t imm_msbs = (Imm & 0xfe0) << 25; |
There was a problem hiding this comment.
Should these variable names be capitalized without underscores? I'm not sure what the conventions for compiler-rt are.
There was a problem hiding this comment.
I capitalized them in order to be more consistent with other places in RISCV XRay.
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| // addi t1, t1, %lo(__xray_FunctionEntry/Exit) | ||
| // lui a0, %hi(function_id) | ||
| // addi a0, a0, %lo(function_id) ;pass function id | ||
| // jalr t1 ;call Tracing hook |
There was a problem hiding this comment.
Can we use ra instead of t1?
There was a problem hiding this comment.
Done. This shrinks the size of sled by 8 bytes.
|
|
||
| .macro SAVE_ARG_REGISTERS | ||
| // Push return registers to stack | ||
| addi sp, sp, -136 |
There was a problem hiding this comment.
The stack pointer must always be 16 byte aligned.
|
|
||
| // Load the handler function pointer into a2 | ||
| la a2, ASM_SYMBOL(_ZN6__xray19XRayPatchedFunctionE) | ||
| ld a2, 0(a2) |
There was a problem hiding this comment.
File header says this code is shared with RV32, but ld isn't valid for RV32.
There was a problem hiding this comment.
Oops, it's fixed now.
|
|
||
| .macro SAVE_ARG_REGISTERS | ||
| // Push argument registers to stack | ||
| addi sp, sp, -100 |
There was a problem hiding this comment.
stack pointer must always be 16 byte aligned.
Use defined (__riscv) && (__riscv_xlen == 32/64) instead.
lenary
reviewed
Dec 4, 2024
And use CFA macros provided by sanitizer_common
topperc
reviewed
Dec 4, 2024
topperc
reviewed
Dec 4, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
|
|
||
| static inline uint32_t encodeJTypeInstruction(uint32_t Opcode, uint32_t Rd, | ||
| uint32_t Imm) { | ||
| uint32_t ImmMSB = (Imm & 0x100000) << 31; |
There was a problem hiding this comment.
Most of these shift amounts are wrong too
There was a problem hiding this comment.
Actually I think they're all wrong, but that would mean the jump in the disable case jumps twice as far as it should. How was that not caught it testing?
There was a problem hiding this comment.
Actually I think they're all wrong
Oops, you're right. I'll fix them shortly.
How was that not caught it testing?
Because the only time we hit the disable case is when these sleds are being unpatched -- something XRay doesn't do by default. The program has to manually call __xray_unpatch to do the unpatching.
There was a problem hiding this comment.
Actually I think they're all wrong, but that would mean the jump in the disable case jumps twice as far as it should. How was that not caught it testing?
It's fixed now. I'd also checked it with tests like compiler-rt/test/xray/TestCases/Posix/patching-unpatching.cpp which should exercise the unpatching logics.
topperc
reviewed
Dec 4, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| // xray_sled_n (32-bit): | ||
| // addi sp, sp, -16 ;create stack frame | ||
| // sw ra, 12(sp) ;save return address | ||
| // sw t1, 8(sp) ;save register t1 |
There was a problem hiding this comment.
Do we need to save t1? It's not used elsewhere in the sled
There was a problem hiding this comment.
Previously it was preserved to share more instructions with RV64's (sled emitting) logics. I just changed it to save more bytes from RV32's sled.
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
|
|
||
| static inline uint32_t encodeJTypeInstruction(uint32_t Opcode, uint32_t Rd, | ||
| uint32_t Imm) { | ||
| uint32_t ImmMSB = (Imm & 0x100000) << 31; |
There was a problem hiding this comment.
Actually I think they're all wrong, but that would mean the jump in the disable case jumps twice as far as it should. How was that not caught it testing?
topperc
reviewed
Dec 4, 2024
topperc
reviewed
Dec 4, 2024
topperc
reviewed
Dec 4, 2024
topperc
reviewed
Dec 4, 2024
topperc
reviewed
Dec 10, 2024
compiler-rt/lib/xray/xray_riscv.cpp
Outdated
| // | ||
| // xray_sled_n: | ||
| // J .tmpN | ||
| // 25 or 33 C.NOPs (50 or 66 bytes) |
topperc
reviewed
Dec 10, 2024
| } | ||
| // XRay uses C.JAL + 25 or 33 C.NOP for each sled in RV32 and RV64, | ||
| // respectively. | ||
| return STI.is64Bit() ? 68 : 52; |
There was a problem hiding this comment.
Good catch. It's fixed now.
topperc
reviewed
Dec 10, 2024
Fixed outdated numebr of NOPs in RV32.
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/153/builds/17205 Here is the relevant piece of the build log for the reference |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add RISC-V support for XRay. The RV64 implementation has been tested in both QEMU and in our hardware environment.
Currently this requires D and C extensions, but since both RV64GC and RVA22/RVA23 are becoming mainstream, I don't think this requirement will be a big problem.
Based on the previous work by @a-poduval : https://reviews.llvm.org/D117929