[libc] implement vdso #91572

SchrodingerZhu · 2024-05-09T08:45:52Z

This PR introduces a mechanism to load symbols from vDSO ¹ on linux platforms. Symbols inside vDSO are typically related to timing API, but may also include things like rt_sigreturn or flush_icache depending on the microarchitecture. Using vDSO for them can potentially avoid the overhead of syscall (not always, a vDSO symbol may fallback to use syscall).

vDSO is an ELF file automatically loaded by OS for the userspace program. To load symbols, we need to do some works as an ELF loader. I believe most libc implementations ²³⁴ do have separate logic for vDSO loading along side normal loaders. So, let's also do it.

We basically follow the same procedure as it is in ³. However, we additionally check the symbol versions⁵ for future-proof.

llvmbot · 2024-05-09T08:46:28Z

@llvm/pr-subscribers-libc

Author: Schrodinger ZHU Yifan (SchrodingerZhu)

Changes

Full diff: https://github.com/llvm/llvm-project/pull/91572.diff

13 Files Affected:

(added) libc/src/__support/OSUtil/fuchsia/vdso.h (+12)
(modified) libc/src/__support/OSUtil/linux/CMakeLists.txt (+16)
(modified) libc/src/__support/OSUtil/linux/aarch64/CMakeLists.txt (+9)
(added) libc/src/__support/OSUtil/linux/aarch64/vdso.h (+47)
(modified) libc/src/__support/OSUtil/linux/arm/CMakeLists.txt (+9)
(added) libc/src/__support/OSUtil/linux/arm/vdso.h (+36)
(modified) libc/src/__support/OSUtil/linux/riscv/CMakeLists.txt (+9)
(added) libc/src/__support/OSUtil/linux/riscv/vdso.h (+55)
(added) libc/src/__support/OSUtil/linux/vdso.cpp (+196)
(added) libc/src/__support/OSUtil/linux/vdso.h (+29)
(modified) libc/src/__support/OSUtil/linux/x86_64/CMakeLists.txt (+9)
(added) libc/src/__support/OSUtil/linux/x86_64/vdso.h (+41)
(added) libc/src/__support/OSUtil/vdso.h (+18)

diff --git a/libc/src/__support/OSUtil/fuchsia/vdso.h b/libc/src/__support/OSUtil/fuchsia/vdso.h
new file mode 100644
index 0000000000000..177d29f879693
--- /dev/null
+++ b/libc/src/__support/OSUtil/fuchsia/vdso.h
@@ -0,0 +1,12 @@
+//===------------- Fuchsia VDSO Header --------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIBC_SRC___SUPPORT_OSUTIL_FUCHSIA_VDSO_H
+#define LLVM_LIBC_SRC___SUPPORT_OSUTIL_FUCHSIA_VDSO_H
+/// TODO: implement fuchsia VDSO
+/// https://fuchsia.dev/fuchsia-src/concepts/kernel/vdso
+#endif // LLVM_LIBC_SRC___SUPPORT_OSUTIL_FUCHSIA_VDSO_H
diff --git a/libc/src/__support/OSUtil/linux/CMakeLists.txt b/libc/src/__support/OSUtil/linux/CMakeLists.txt
index 239d115704927..e7411c14711d2 100644
--- a/libc/src/__support/OSUtil/linux/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/linux/CMakeLists.txt
@@ -16,3 +16,19 @@ add_object_library(
     libc.src.__support.common
     libc.src.__support.CPP.string_view
 )
+
+add_object_library(
+  vdso
+  HDRS
+    vdso.h
+  SRCS
+    vdso.cpp
+  DEPENDS
+    .${LIBC_TARGET_ARCHITECTURE}.vdso
+    libc.src.__support.CPP.array
+    libc.src.__support.CPP.string_view
+    libc.src.__support.threads.callonce
+    libc.src.__support.threads.linux.futex_word_type
+    libc.src.errno.errno
+    libc.src.sys.auxv.getauxval
+)
diff --git a/libc/src/__support/OSUtil/linux/aarch64/CMakeLists.txt b/libc/src/__support/OSUtil/linux/aarch64/CMakeLists.txt
index eea9badc46cae..a574675535694 100644
--- a/libc/src/__support/OSUtil/linux/aarch64/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/linux/aarch64/CMakeLists.txt
@@ -5,3 +5,12 @@ add_header_library(
   DEPENDS
     libc.src.__support.common
 )
+
+add_header_library(
+  vdso
+  HDRS
+    vdso.h
+  DEPENDS
+    libc.src.__support.common
+    lubc.src.__support.CPP.string_view
+)
diff --git a/libc/src/__support/OSUtil/linux/aarch64/vdso.h b/libc/src/__support/OSUtil/linux/aarch64/vdso.h
new file mode 100644
index 0000000000000..55024e61ee914
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/aarch64/vdso.h
@@ -0,0 +1,47 @@
+//===---------- aarch64 vdso configuration ------------------------* C++ *-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "src/__support/CPP/string_view.h"
+namespace LIBC_NAMESPACE {
+namespace vdso {
+// macro definitions
+#define LIBC_VDSO_HAS_RT_SIGRETURN
+#define LIBC_VDSO_HAS_GETTIMEOFDAY
+#define LIBC_VDSO_HAS_CLOCK_GETTIME
+#define LIBC_VDSO_HAS_CLOCK_GETRES
+
+// list of VDSO symbols
+enum class VDSOSym {
+  RTSigReturn,
+  GetTimeOfDay,
+  ClockGetTime,
+  ClockGetRes,
+  VDSOSymCount
+};
+
+// translate VDSOSym to symbol names
+LIBC_INLINE constexpr cpp::string_view symbol_name(VDSOSym sym) {
+  switch (sym) {
+  case VDSOSym::RTSigReturn:
+    return "__kernel_rt_sigreturn";
+  case VDSOSym::GetTimeOfDay:
+    return "__kernel_gettimeofday";
+  case VDSOSym::ClockGetTime:
+    return "__kernel_clock_gettime";
+  case VDSOSym::ClockGetRes:
+    return "__kernel_clock_getres";
+  default:
+    return "";
+  }
+}
+
+// symbol versions
+LIBC_INLINE constexpr cpp::string_view symbol_version(VDSOSym) {
+  return "LINUX_2.6.39";
+}
+} // namespace vdso
+} // namespace LIBC_NAMESPACE
diff --git a/libc/src/__support/OSUtil/linux/arm/CMakeLists.txt b/libc/src/__support/OSUtil/linux/arm/CMakeLists.txt
index 733366f6d4a2e..5f82762a5add3 100644
--- a/libc/src/__support/OSUtil/linux/arm/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/linux/arm/CMakeLists.txt
@@ -5,3 +5,12 @@ add_header_library(
   DEPENDS
     libc.src.__support.common
 )
+
+add_header_library(
+  vdso
+  HDRS
+    vdso.h
+  DEPENDS
+    libc.src.__support.common
+    lubc.src.__support.CPP.string_view
+)
diff --git a/libc/src/__support/OSUtil/linux/arm/vdso.h b/libc/src/__support/OSUtil/linux/arm/vdso.h
new file mode 100644
index 0000000000000..5d9ff5d929ab6
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/arm/vdso.h
@@ -0,0 +1,36 @@
+//===---------- arm vdso configuration ----------------------------* C++ *-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "src/__support/CPP/string_view.h"
+namespace LIBC_NAMESPACE {
+namespace vdso {
+// macro definitions
+#define LIBC_VDSO_HAS_GETTIMEOFDAY
+#define LIBC_VDSO_HAS_CLOCK_GETTIME
+
+// list of VDSO symbols
+enum class VDSOSym {
+  GetTimeOfDay,
+  ClockGetTime,
+};
+
+// translate VDSOSym to symbol names
+LIBC_INLINE constexpr cpp::string_view symbol_name(VDSOSym sym) {
+  switch (sym) {
+  case VDSOSym::GetTimeOfDay:
+    return "__vdso_gettimeofday";
+  case VDSOSym::ClockGetTime:
+    return "__vdso_clock_gettime";
+  }
+}
+
+// symbol versions
+LIBC_INLINE constexpr cpp::string_view symbol_version(VDSOSym) {
+  return "LINUX_2.6";
+}
+} // namespace vdso
+} // namespace LIBC_NAMESPACE
diff --git a/libc/src/__support/OSUtil/linux/riscv/CMakeLists.txt b/libc/src/__support/OSUtil/linux/riscv/CMakeLists.txt
index e271204f51982..26ff84dacba7c 100644
--- a/libc/src/__support/OSUtil/linux/riscv/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/linux/riscv/CMakeLists.txt
@@ -5,3 +5,12 @@ add_header_library(
   DEPENDS
     libc.src.__support.common
 )
+
+add_header_library(
+  vdso
+  HDRS
+    vdso.h
+  DEPENDS
+    libc.src.__support.common
+    lubc.src.__support.CPP.string_view
+)
diff --git a/libc/src/__support/OSUtil/linux/riscv/vdso.h b/libc/src/__support/OSUtil/linux/riscv/vdso.h
new file mode 100644
index 0000000000000..394c06e6a5a72
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/riscv/vdso.h
@@ -0,0 +1,55 @@
+//===---------- RISC-V vdso configuration -------------------------* C++ *-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "src/__support/CPP/string_view.h"
+namespace LIBC_NAMESPACE {
+namespace vdso {
+// macro definitions
+#define LIBC_VDSO_HAS_RT_SIGRETURN
+#define LIBC_VDSO_HAS_GETTIMEOFDAY
+#define LIBC_VDSO_HAS_CLOCK_GETTIME
+#define LIBC_VDSO_HAS_CLOCK_GETRES
+#define LIBC_VDSO_HAS_GETCPU
+#define LIBC_VDSO_HAS_FLUSH_ICACHE
+
+// list of VDSO symbols
+enum class VDSOSym {
+  RTSigReturn,
+  GetTimeOfDay,
+  ClockGetTime,
+  ClockGetRes,
+  GetCpu,
+  FlushICache,
+  VDSOSymCount
+};
+
+// translate VDSOSym to symbol names
+LIBC_INLINE constexpr cpp::string_view symbol_name(VDSOSym sym) {
+  switch (sym) {
+  case VDSOSym::RTSigReturn:
+    return "__vdso_rt_sigreturn";
+  case VDSOSym::GetTimeOfDay:
+    return "__vdso_gettimeofday";
+  case VDSOSym::ClockGetTime:
+    return "__vdso_clock_gettime";
+  case VDSOSym::ClockGetRes:
+    return "__vdso_clock_getres";
+  case VDSOSym::GetCpu:
+    return "__vdso_getcpu";
+  case VDSOSym::FlushICache:
+    return "__vdso_flush_icache";
+  default:
+    return "";
+  }
+}
+
+// symbol versions
+LIBC_INLINE constexpr cpp::string_view symbol_version(VDSOSym) {
+  return "LINUX_4.15";
+}
+} // namespace vdso
+} // namespace LIBC_NAMESPACE
diff --git a/libc/src/__support/OSUtil/linux/vdso.cpp b/libc/src/__support/OSUtil/linux/vdso.cpp
new file mode 100644
index 0000000000000..298a332b31922
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/vdso.cpp
@@ -0,0 +1,196 @@
+//===------------- Linux VDSO Implementation --------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "src/__support/OSUtil/linux/vdso.h"
+#include "src/__support/CPP/array.h"
+#include "src/__support/CPP/string_view.h"
+#include "src/__support/threads/callonce.h"
+#include "src/__support/threads/linux/futex_word.h"
+#include "src/errno/libc_errno.h"
+#include "src/sys/auxv/getauxval.h"
+#include <linux/auxvec.h>
+#include <linux/elf.h>
+
+#ifndef ElfW
+#if __POINTER_WIDTH__ == 32
+#define ElfW(type) Elf32_##type
+#else
+#define ElfW(type) Elf64_##type
+#endif
+#endif
+
+namespace LIBC_NAMESPACE {
+namespace vdso {
+
+// See https://refspecs.linuxfoundation.org/LSB_1.3.0/gLSB/gLSB/symverdefs.html
+struct Verdaux {
+  ElfW(Word) vda_name; /* Version or dependency names */
+  ElfW(Word) vda_next; /* Offset in bytes to next verdaux
+                          entry */
+};
+struct Verdef {
+  ElfW(Half) vd_version; /* Version revision */
+  ElfW(Half) vd_flags;   /* Version information */
+  ElfW(Half) vd_ndx;     /* Version Index */
+  ElfW(Half) vd_cnt;     /* Number of associated aux entries */
+  ElfW(Word) vd_hash;    /* Version name hash value */
+  ElfW(Word) vd_aux;     /* Offset in bytes to verdaux array */
+  ElfW(Word) vd_next;    /* Offset in bytes to next verdef
+                            entry */
+  Verdef *next() const {
+    return reinterpret_cast<Verdef *>(reinterpret_cast<uintptr_t>(this) +
+                                      vd_next);
+  }
+  Verdaux *aux() const {
+    return reinterpret_cast<Verdaux *>(reinterpret_cast<uintptr_t>(this) +
+                                       vd_aux);
+  }
+};
+
+// version search procedure specified by
+// https://refspecs.linuxfoundation.org/LSB_1.3.0/gLSB/gLSB/symversion.html#SYMVERTBL
+cpp::string_view find_version(Verdef *verdef, ElfW(Half) * versym,
+                              const char *strtab, size_t idx) {
+  static constexpr ElfW(Half) VER_FLG_BASE = 0x1;
+  ElfW(Half) identifier = versym[idx] & 0x7FFF;
+  // iterate through all version definitions
+  for (Verdef *def = verdef; def->vd_next != 0; def = def->next()) {
+    // skip if this is a file-level version
+    if (def->vd_flags & VER_FLG_BASE)
+      continue;
+    // check if the version identifier matches
+    if ((def->vd_flags & 0x7FFF) == identifier) {
+      Verdaux *aux = def->aux();
+      return strtab + aux->vda_name;
+    }
+  }
+  return "";
+}
+
+using VDSOArray =
+    cpp::array<void *, static_cast<size_t>(VDSOSym::VDSOSymCount)>;
+
+static VDSOArray symbol_table;
+
+void *get_symbol(VDSOSym sym) {
+  // if sym is invalid, return nullptr
+  const size_t index = static_cast<size_t>(sym);
+  if (index >= symbol_table.size())
+    return nullptr;
+
+  static FutexWordType once_flag = 0;
+  callonce(reinterpret_cast<CallOnceFlag *>(&once_flag), [] {
+    // first clear the symbol table
+    for (auto &i : symbol_table) {
+      i = nullptr;
+    }
+
+    // get the address of the VDSO, protect errno since getauxval may change it
+    int errno_backup = libc_errno;
+    uintptr_t vdso_ehdr_addr = getauxval(AT_SYSINFO_EHDR);
+    // Get the memory address of the vDSO ELF header.
+    auto vdso_ehdr = reinterpret_cast<ElfW(Ehdr) *>(vdso_ehdr_addr);
+    // leave the table unpopulated if we don't have vDSO
+    if (vdso_ehdr == nullptr) {
+      libc_errno = errno_backup;
+      return;
+    }
+
+    // count entries
+    size_t symbol_count = 0;
+    // locate the section header inside the elf using the section header offset
+    auto vdso_shdr =
+        reinterpret_cast<ElfW(Shdr) *>(vdso_ehdr_addr + vdso_ehdr->e_shoff);
+    // iterate all sections until we locate the dynamic symbol section
+    for (size_t i = 0; i < vdso_ehdr->e_shnum; ++i) {
+      if (vdso_shdr[i].sh_type == SHT_DYNSYM) {
+        // dynamic symbol section is a table section
+        // therefore, the number of entries can be computed as the ratio
+        // of the section size to the size of a single entry
+        symbol_count = vdso_shdr[i].sh_size / vdso_shdr[i].sh_entsize;
+        break;
+      }
+    }
+
+    // early return if no symbol is found
+    if (symbol_count == 0)
+      return;
+
+    // We need to find both the loadable segment and the dynamic linking of the
+    // vDSO.
+    auto vdso_addr = static_cast<ElfW(Addr)>(-1);
+    ElfW(Dyn) *vdso_dyn = nullptr;
+    // compute vdso_phdr as the program header using the program header offset
+    ElfW(Phdr) *vdso_phdr =
+        reinterpret_cast<ElfW(Phdr) *>(vdso_ehdr_addr + vdso_ehdr->e_phoff);
+    // iterate through all the program headers until we get the desired pieces
+    for (size_t i = 0; i < vdso_ehdr->e_phnum; ++i) {
+      if (vdso_phdr[i].p_type == PT_DYNAMIC)
+        vdso_dyn = reinterpret_cast<ElfW(Dyn) *>(vdso_ehdr_addr +
+                                                 vdso_phdr[i].p_offset);
+
+      if (vdso_phdr[i].p_type == PT_LOAD)
+        vdso_addr =
+            vdso_ehdr_addr + vdso_phdr[i].p_offset - vdso_phdr[i].p_vaddr;
+
+      if (vdso_addr && vdso_dyn)
+        break;
+    }
+    // early return if either the dynamic linking or the loadable segment is not
+    // found
+    if (vdso_dyn == nullptr || vdso_addr == static_cast<ElfW(Addr)>(-1))
+      return;
+
+    // now, locate several more tables inside the dynmaic linking section
+    const char *strtab = nullptr;
+    ElfW(Sym) *symtab = nullptr;
+    ElfW(Half) *versym = nullptr;
+    Verdef *verdef = nullptr;
+    for (ElfW(Dyn) *d = vdso_dyn; d->d_tag != DT_NULL; ++d) {
+      switch (d->d_tag) {
+      case DT_STRTAB:
+        strtab = reinterpret_cast<const char *>(vdso_addr + d->d_un.d_ptr);
+        break;
+      case DT_SYMTAB:
+        symtab = reinterpret_cast<ElfW(Sym) *>(vdso_addr + d->d_un.d_ptr);
+        break;
+      case DT_VERSYM:
+        versym = reinterpret_cast<uint16_t *>(vdso_addr + d->d_un.d_ptr);
+        break;
+      case DT_VERDEF:
+        verdef = reinterpret_cast<Verdef *>(vdso_addr + d->d_un.d_ptr);
+        break;
+      }
+      if (strtab && symtab && versym && verdef) {
+        break;
+      }
+    }
+    if (strtab == nullptr || symtab == nullptr)
+      return;
+
+    for (size_t i = 0; i < symbol_table.size(); ++i) {
+      for (size_t j = 0; j < symbol_count; ++j) {
+        auto sym = static_cast<VDSOSym>(i);
+        if (symbol_name(sym) == strtab + symtab[j].st_name) {
+          // we find a symbol with desired name
+          // now we need to check if it has the right version
+          if (versym && verdef)
+            if (symbol_version(sym) != find_version(verdef, versym, strtab, j))
+              continue;
+
+          // put the symbol address into the symbol table
+          symbol_table[i] =
+              reinterpret_cast<void *>(vdso_addr + symtab[j].st_value);
+        }
+      }
+    }
+  });
+
+  return symbol_table[index];
+}
+} // namespace vdso
+} // namespace LIBC_NAMESPACE
diff --git a/libc/src/__support/OSUtil/linux/vdso.h b/libc/src/__support/OSUtil/linux/vdso.h
new file mode 100644
index 0000000000000..2d71db13303f0
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/vdso.h
@@ -0,0 +1,29 @@
+//===------------- Linux VDSO Header ----------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIBC_SRC___SUPPORT_OSUTIL_LINUX_VDSO_H
+#define LLVM_LIBC_SRC___SUPPORT_OSUTIL_LINUX_VDSO_H
+#include "src/__support/common.h"
+#include "src/__support/macros/properties/architectures.h"
+
+#if defined(LIBC_TARGET_ARCH_IS_X86)
+#include "x86_64/vdso.h"
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#include "aarch64/vdso.h"
+#elif defined(LIBC_TARGET_ARCH_IS_ARM)
+#include "arm/vdso.h"
+#elif defined(LIBC_TARGET_ARCH_IS_RISCV)
+#include "riscv/vdso.h"
+#endif
+
+namespace LIBC_NAMESPACE {
+namespace vdso {
+void *get_symbol(VDSOSym);
+} // namespace vdso
+
+} // namespace LIBC_NAMESPACE
+#endif // LLVM_LIBC_SRC___SUPPORT_OSUTIL_LINUX_VDSO_H
diff --git a/libc/src/__support/OSUtil/linux/x86_64/CMakeLists.txt b/libc/src/__support/OSUtil/linux/x86_64/CMakeLists.txt
index a7f2d74e6353e..83f5eb87f8180 100644
--- a/libc/src/__support/OSUtil/linux/x86_64/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/linux/x86_64/CMakeLists.txt
@@ -5,3 +5,12 @@ add_header_library(
   DEPENDS
     libc.src.__support.common
 )
+
+add_header_library(
+  vdso
+  HDRS
+    vdso.h
+  DEPENDS
+    libc.src.__support.common
+    lubc.src.__support.CPP.string_view
+)
diff --git a/libc/src/__support/OSUtil/linux/x86_64/vdso.h b/libc/src/__support/OSUtil/linux/x86_64/vdso.h
new file mode 100644
index 0000000000000..13b8e3d8cc6f4
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/x86_64/vdso.h
@@ -0,0 +1,41 @@
+//===---------- x86/x86_64 vdso configuration ---------------------* C++ *-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "src/__support/CPP/string_view.h"
+namespace LIBC_NAMESPACE {
+namespace vdso {
+// macro definitions
+#define LIBC_VDSO_HAS_CLOCK_GETTIME
+#define LIBC_VDSO_HAS_GETCPU
+#define LIBC_VDSO_HAS_GETTIMEOFDAY
+#define LIBC_VDSO_HAS_TIME
+
+// list of VDSO symbols
+enum class VDSOSym { ClockGetTime, GetCpu, GetTimeOfDay, Time, VDSOSymCount };
+
+// translate VDSOSym to symbol names
+LIBC_INLINE constexpr cpp::string_view symbol_name(VDSOSym sym) {
+  switch (sym) {
+  case VDSOSym::ClockGetTime:
+    return "__vdso_clock_gettime";
+  case VDSOSym::GetCpu:
+    return "__vdso_getcpu";
+  case VDSOSym::GetTimeOfDay:
+    return "__vdso_gettimeofday";
+  case VDSOSym::Time:
+    return "__vdso_time";
+  default:
+    return "";
+  }
+}
+
+// symbol versions
+LIBC_INLINE constexpr cpp::string_view symbol_version(VDSOSym) {
+  return "LINUX_2.6";
+}
+} // namespace vdso
+} // namespace LIBC_NAMESPACE
diff --git a/libc/src/__support/OSUtil/vdso.h b/libc/src/__support/OSUtil/vdso.h
new file mode 100644
index 0000000000000..93bbc98da19b4
--- /dev/null
+++ b/libc/src/__support/OSUtil/vdso.h
@@ -0,0 +1,18 @@
+//===--------------- Virtual DSO Support ------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_OSUTIL_VDSO_H
+#define LLVM_LIBC_SRC___SUPPORT_OSUTIL_VDSO_H
+
+#if defined(__linux__)
+#include "linux/vdso.h"
+#elif defined(__Fuchsia__)
+#include "fuchsia/vdso.h"
+#endif
+
+#endif // LLVM_LIBC_SRC___SUPPORT_OSUTIL_VDSO_H

libc/src/__support/OSUtil/linux/aarch64/vdso.h

SchrodingerZhu · 2024-05-09T16:32:59Z

strace ./projects/libc/test/src/__support/OSUtil/linux/libc.test.src.__support.OSUtil.linux.vdso_test.__hermetic__.__build__

On x86, the above line should generate no syscall for gettimeofday, which is indeed the case:

execve("./projects/libc/test/src/__support/OSUtil/linux/libc.test.src.__support.OSUtil.linux.vdso_test.__hermetic__.__build__", ["./projects/libc/test/src/__suppo"...], 0x7fffea955d20 /* 45 vars */) = 0
gettid()                                = 2701205
mmap(NULL, 41064, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7a9d7cdd6000
getrandom("\xca\x7c\xd2\xa3\x43\x65\x83\xfb", 8, 0) = 8
arch_prctl(ARCH_SET_FS, 0x7a9d7cde0038) = 0
write(2, "\33[32m", 5)                  = 5
write(2, "[ RUN      ] ", 13[ RUN      ] )           = 13
write(2, "\33[0m", 4)                   = 4
write(2, "LlvmLibcOSUtilVDSOTest.SymbolsDe"..., 37LlvmLibcOSUtilVDSOTest.SymbolsDefined) = 37
write(2, "\n", 1
)                       = 1
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=696960}) = 0
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=718680}) = 0
write(2, "\33[32m", 5)                  = 5
write(2, "[       OK ] ", 13[       OK ] )           = 13
write(2, "\33[0m", 4)                   = 4
write(2, "LlvmLibcOSUtilVDSOTest.SymbolsDe"..., 37LlvmLibcOSUtilVDSOTest.SymbolsDefined) = 37
write(2, " (took ", 7 (took )                  = 7
write(2, "22", 222)                       = 2
write(2, " us)\n", 5 us)
)                   = 5
write(2, "\33[32m", 5)                  = 5
write(2, "[ RUN      ] ", 13[ RUN      ] )           = 13
write(2, "\33[0m", 4)                   = 4
write(2, "LlvmLibcOSUtilVDSOTest.GetTimeOf"..., 35LlvmLibcOSUtilVDSOTest.GetTimeOfDay) = 35
write(2, "\n", 1
)                       = 1
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=863630}) = 0
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=883420}) = 0
write(2, "\33[32m", 5)                  = 5
write(2, "[       OK ] ", 13[       OK ] )           = 13
write(2, "\33[0m", 4)                   = 4
write(2, "LlvmLibcOSUtilVDSOTest.GetTimeOf"..., 35LlvmLibcOSUtilVDSOTest.GetTimeOfDay) = 35
write(2, " (took ", 7 (took )                  = 7
write(2, "20", 220)                       = 2
write(2, " us)\n", 5 us)
)                   = 5
write(2, "Ran ", 4Ran )                     = 4
write(2, "2", 12)                        = 1
write(2, " tests. ", 8 tests. )                 = 8
write(2, " PASS: ", 7 PASS: )                  = 7
write(2, "2", 12)                        = 1
write(2, " ", 1 )                        = 1
write(2, " FAIL: ", 7 FAIL: )                  = 7
write(2, "0", 10)                        = 1
write(2, "\n", 1
)                       = 1
munmap(0x7a9d7cde0038, 41064)           = -1 EINVAL (Invalid argument)
exit_group(0)                           = ?
+++ exited with 0 +++

SchrodingerZhu · 2024-05-09T19:24:01Z

current build error should be addressed in #91485

michaelrj-google · 2024-05-09T23:14:55Z

This patch overall LGTM, I'll review it more thoroughly once the build issue is figured out

SchrodingerZhu · 2024-05-10T20:26:31Z

I collapse the overlay build problem fix due to CI's failure to checkout.

SchrodingerZhu · 2024-05-10T20:53:14Z

@michaelrj-google CI build should be working now.

However, there is a complication:

In seems that the object library in OSUtil is not collected into the static library in #91805 . I am looking into it.

It seems that the object library of header library will not be collected. I removed the toplevel header library for now.

libc/src/__support/OSUtil/linux/aarch64/vdso.h

libc/src/__support/OSUtil/linux/vdso.cpp

libc/src/__support/OSUtil/linux/vdso.h

libc/src/__support/OSUtil/linux/riscv/vdso.h

nickdesaulniers

Also, I guess this does not yet use vdso::get_symbol to implement any of the functions; it just lays the groundwork for doing so.

SchrodingerZhu · 2024-05-22T03:17:37Z

Also, I guess this does not yet use vdso::get_symbol to implement any of the functions; it just lays the groundwork for doing so.

Yes, there is a following up patch to migrate clock_gettime to vDSO. I guess we can open a bunch of GFIs after the vDSO patch.

jyknight · 2024-05-22T03:48:03Z

The manpage you reference is incomplete w.r.t. which functions are supported on which architectures. At least, x86-64, it's missing clock_getres, which was added in 2019.

You might wish to look at the kernel sources directly, instead of this outdated documentation -- the info is in the "VERSION" section in the vdso linker scripts. E.g. less $(git grep -l VERSION arch/**/*lds.S)

libc/src/__support/OSUtil/linux/vdso.cpp

nickdesaulniers · 2024-05-28T18:44:46Z

The manpage you reference is incomplete w.r.t. which functions are supported on which architectures. At least, x86-64, it's missing clock_getres, which was added in 2019.

You might wish to look at the kernel sources directly, instead of this outdated documentation -- the info is in the "VERSION" section in the vdso linker scripts. E.g. less $(git grep -l VERSION arch/**/*lds.S)

Right, this was what I was afraid of; man pages getting out of date. The kernel sources are the source of truth, so I'd prefer to have comments in our sources linking to the relevant sources from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/. In particular, there's active discussion on LKML around adding a new symbol to vdso; getrandom.

libc/hdr/CMakeLists.txt

SchrodingerZhu · 2024-09-10T17:08:50Z

@lntue fixed

libc/src/__support/OSUtil/linux/vdso.cpp

SchrodingerZhu · 2024-09-10T18:05:08Z

fixed

michaelrj-google · 2024-09-10T18:14:26Z

libc/src/__support/OSUtil/linux/aarch64/vdso.h

+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIBC_SRC___SUPPORT_OSUTIL_LINUX_AARCH64_VDSO_H
+#define LLVM_LIBC_SRC___SUPPORT_OSUTIL_LINUX_AARCH64_VDSO_H
+#include "src/__support/CPP/string_view.h"


you should also include the top level vdso header (src/__support/OSUtil/linux/vdso.h). Each header should be able to build on its own.

That would make a recursive include chain, which is not good. (it wouldn't actually infinitely recurse because of header guards, but it's still a bad idea).

Perhaps instead these arch/vdso.h files should be renamed to ".inc" and be without header guards, to show that they're not intended to be standalone headers?

Or we can refactor VDSOSym to its own header file vdsosym.h to be included in all targets and the main vdso.h.

fixed in 3031d4a

llvmbot added the libc label May 9, 2024

SchrodingerZhu marked this pull request as draft May 9, 2024 09:02

SchrodingerZhu force-pushed the libc/vdso-2 branch from 6c012a1 to 3c30e8f Compare May 9, 2024 09:08

SchrodingerZhu marked this pull request as ready for review May 9, 2024 09:08

SchrodingerZhu requested a review from nickdesaulniers May 9, 2024 09:09

lntue reviewed May 9, 2024

View reviewed changes

libc/src/__support/OSUtil/linux/aarch64/vdso.h Outdated Show resolved Hide resolved

lntue requested review from frobtech and michaelrj-google May 9, 2024 14:39

SchrodingerZhu force-pushed the libc/vdso-2 branch 2 times, most recently from 6edd8e8 to 0341edb Compare May 9, 2024 16:22

SchrodingerZhu force-pushed the libc/vdso-2 branch from 2b516e6 to d94699a Compare May 9, 2024 18:20

SchrodingerZhu force-pushed the libc/vdso-2 branch 2 times, most recently from 6eec45c to e3212e7 Compare May 10, 2024 20:24

SchrodingerZhu mentioned this pull request May 10, 2024

[libc] update clock_gettime implementation with vDSO #91805

Closed

SchrodingerZhu force-pushed the libc/vdso-2 branch from ed9ac31 to 379ade6 Compare May 13, 2024 21:35

nickdesaulniers reviewed May 21, 2024

View reviewed changes

SchrodingerZhu force-pushed the libc/vdso-2 branch 2 times, most recently from 1d16d96 to c23dde8 Compare May 26, 2024 20:31

SchrodingerZhu requested a review from nickdesaulniers May 28, 2024 17:58

nickdesaulniers reviewed May 28, 2024

View reviewed changes

libc/src/__support/OSUtil/linux/vdso.cpp Outdated Show resolved Hide resolved

libc/src/__support/OSUtil/linux/vdso.cpp Outdated Show resolved Hide resolved

Yifan Zhu added 4 commits September 10, 2024 09:43

remove extra friend

6478047

fix riscv

3fefec5

fix typo

a275cb8

add typed symbol

db4ccf5

SchrodingerZhu force-pushed the libc/vdso-2 branch from 7775de0 to db4ccf5 Compare September 10, 2024 13:44

SchrodingerZhu added 6 commits September 10, 2024 10:21

address CR and latest changes

2e158fc

address CR and latest changes

cca6647

simplify dispatcher

9017884

fix format

56fe811

fix format

18b0fd9

fix format

df58616

SchrodingerZhu requested a review from lntue September 10, 2024 15:10