Merge branch 'main' into aligned_alloc

Author: PiJoules

bolt/docs/OptimizingLinux.md

@@ -0,0 +1,120 @@
+# Optimizing Linux Kernel with BOLT
+
+
+## Introduction
+
+Many Linux applications spend a significant amount of their execution time in the kernel. Thus, when we consider code optimization for system performance, it is essential to improve the CPU utilization not only in the user-space applications and libraries but also in the kernel. BOLT has demonstrated double-digit gains while being applied to user-space programs. This guide shows how to apply BOLT to the x86-64 Linux kernel and enhance your system's performance. In our experiments, BOLT boosted database TPS by 2 percent when applied to the kernel compiled with the highest level optimizations, including PGO and LTO. The database spent ~40% of the time in the kernel and was quite sensitive to kernel performance.
+
+BOLT optimizes code layout based on a low-level execution profile collected with the Linux `perf` tool. The best quality profile should include branch history, such as Intel's last branch records (LBR). BOLT runs on a linked binary and reorders the code while combining frequently executed blocks of instructions in a manner best suited for the hardware. Other than branch instructions, most of the code is left unchanged. Additionally, BOLT updates all metadata associated with the modified code, including DWARF debug information and Linux ORC unwind information.
+
+While BOLT optimizations are not specific to the Linux kernel, certain quirks distinguish the kernel from user-level applications.
+
+BOLT has been successfully applied to and tested with several flavors of the x86-64 Linux kernel.
+
+
+## QuickStart Guide
+
+BOLT operates on a statically-linked kernel executable, a.k.a. `vmlinux` binary. However, most Linux distributions use a `vmlinuz` compressed image for system booting. To use BOLT on the kernel, you must either repackage `vmlinuz` after BOLT optimizations or add steps for running BOLT into the kernel build and rebuild `vmlinuz`. Uncompressing `vmlinuz` and repackaging it with a new `vmlinux` binary falls beyond the scope of this guide, and at some point, we may add the capability to run BOLT directly on `vmlinuz`. Meanwhile, this guide focuses on steps for integrating BOLT into the kernel build process.
+
+
+### Building the Kernel
+
+After downloading the kernel sources and configuration for your distribution, you should be able to build `vmlinuz` using the `make bzImage` command. Ideally, the kernel should binary match the kernel on the system you are about to optimize (the target system). The binary matching part is critical as BOLT performs profile matching and optimizations at the binary level. We recommend installing a freshly built kernel on the target system to avoid any discrepancies.
+
+Note that the kernel build will produce several artifacts besides bzImage. The most important of them is the uncompressed `vmlinux` binary, which will be used in the next steps. Make sure to save this file.
+
+Build and target systems should have a `perf` tool installed for collecting and processing profiles. If your build system differs from the target, make sure `perf` versions are compatible. The build system should also have the latest BOLT binary and tools (`llvm-bolt`, `perf2bolt`).
+
+Once the target system boots with the freshly-built kernel, start your workload, such as a database benchmark. While the system is under load, collect the kernel profile using perf:
+
+
+```bash
+$ sudo perf record -a -e cycles -j any,k -F 5000 -- sleep 600
+```
+
+
+Convert `perf` profile into a format suitable for BOLT passing the `vmlinux` binary to `perf2bolt`:
+
+
+```bash
+$ sudo chwon $USER perf.data
+$ perf2bolt -p perf.data -o perf.fdata vmlinux
+```
+
+
+Under a high load, `perf.data` should be several gigabytes in size and you should expect the converted `perf.fdata` not to exceed 100 MB.
+
+Two changes are required for the kernel build. The first one is optional but highly recommended. It introduces a BOLT-reserved space into `vmlinux` code section:
+
+
+```diff
+--- a/arch/x86/kernel/vmlinux.lds.S
++++ b/arch/x86/kernel/vmlinux.lds.S
+@@ -139,6 +139,11 @@ SECTIONS
+                STATIC_CALL_TEXT
+                *(.gnu.warning)
+
++    /* Allocate space for BOLT */
++    __bolt_reserved_start = .;
++               . += 2048 * 1024;
++    __bolt_reserved_end = .;
++
+ #ifdef CONFIG_RETPOLINE
+                __indirect_thunk_start = .;
+                *(.text.__x86.*)
+```
+
+
+The second patch adds a step that runs BOLT on `vmlinux` binary:
+
+
+```diff
+--- a/scripts/link-vmlinux.sh
++++ b/scripts/link-vmlinux.sh
+@@ -340,5 +340,13 @@ if is_enabled CONFIG_KALLSYMS; then
+        fi
+ fi
+
++# Apply BOLT
++BOLT=llvm-bolt
++BOLT_PROFILE=perf.fdata
++BOLT_OPTS="--dyno-stats --eliminate-unreachable=0 --reorder-blocks=ext-tsp --simplify-conditional-tail-calls=0 --skip-funcs=__entry_text_start,irq_entries_start --split-functions"
++mv vmlinux vmlinux.pre-bolt
++echo BOLTing vmlinux
++${BOLT} vmlinux.pre-bolt -o vmlinux --data ${BOLT_PROFILE} ${BOLT_OPTS}
++
+ # For fixdep
+ echo "vmlinux: $0" > .vmlinux.d
+```
+
+
+If you skipped the first step or are running BOLT on a pre-built `vmlinux` binary, drop the `--split-functions` option.
+
+
+## Performance Expectations
+
+By improving the code layout, BOLT can boost the kernel's performance by up to 5% by reducing instruction cache misses and branch mispredictions. When measuring total system performance, you should scale this number accordingly based on the time your application spends in the kernel (excluding I/O time).
+
+
+## Profile Quality
+
+The timing and duration of the profiling may have a significant effect on the performance of the BOLTed kernel. If you don't know your workload well, it's recommended that you profile for the whole duration of the benchmark run. As longer times will result in larger `perf.data` files, you can lower the profiling frequency by providing a smaller value of `-F` flag. E.g., to record the kernel profile for half an hour, use the following command:
+
+
+```bash
+$ sudo perf record -a -e cycles -j any,k -F 1000 -- sleep 1800
+```
+
+
+
+## BOLT Disassembly
+
+BOLT annotates the disassembly with control-flow information and attaches Linux-specific metadata to the code. To view annotated disassembly, run:
+
+
+```bash
+$ llvm-bolt vmlinux -o /dev/null --print-cfg
+```
+
+
+If you want to limit the disassembly to a set of functions, add `--print-only=<func1regex>,<func2regex>,...`, where a function name is specified using regular expressions.

clang/include/clang/Driver/Options.td

@@ -5421,7 +5421,6 @@ def module_file_info : Flag<["-"], "module-file-info">, Flags<[]>,
   HelpText<"Provide information about a particular module file">;
 def mthumb : Flag<["-"], "mthumb">, Group<m_Group>;
 def mtune_EQ : Joined<["-"], "mtune=">, Group<m_Group>,
-  Visibility<[ClangOption, FlangOption]>,
   HelpText<"Only supported on AArch64, PowerPC, RISC-V, SPARC, SystemZ, and X86">;
 def multi__module : Flag<["-"], "multi_module">;
 def multiply__defined__unused : Separate<["-"], "multiply_defined_unused">;
@@ -6738,6 +6737,9 @@ def emit_hlfir : Flag<["-"], "emit-hlfir">, Group<Action_Group>,
 
 let Visibility = [CC1Option, CC1AsOption] in {
 
+def tune_cpu : Separate<["-"], "tune-cpu">,
+  HelpText<"Tune for a specific cpu type">,
+  MarshallingInfoString<TargetOpts<"TuneCPU">>;
 def target_abi : Separate<["-"], "target-abi">,
   HelpText<"Target a particular ABI type">,
   MarshallingInfoString<TargetOpts<"ABI">>;
@@ -6764,9 +6766,6 @@ def darwin_target_variant_triple : Separate<["-"], "darwin-target-variant-triple
 
 let Visibility = [CC1Option, CC1AsOption, FC1Option] in {
 
-def tune_cpu : Separate<["-"], "tune-cpu">,
-  HelpText<"Tune for a specific cpu type">,
-  MarshallingInfoString<TargetOpts<"TuneCPU">>;
 def target_cpu : Separate<["-"], "target-cpu">,
   HelpText<"Target a specific cpu type">,
   MarshallingInfoString<TargetOpts<"CPU">>;

clang/lib/Driver/ToolChains/Flang.cpp

@@ -15,7 +15,6 @@
 #include "llvm/Frontend/Debug/Options.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
-#include "llvm/TargetParser/Host.h"
 #include "llvm/TargetParser/RISCVISAInfo.h"
 #include "llvm/TargetParser/RISCVTargetParser.h"
 
@@ -412,13 +411,6 @@ void Flang::addTargetOptions(const ArgList &Args,
   }
 
   // TODO: Add target specific flags, ABI, mtune option etc.
-  if (const Arg *A = Args.getLastArg(options::OPT_mtune_EQ)) {
-    CmdArgs.push_back("-tune-cpu");
-    if (A->getValue() == StringRef{"native"})
-      CmdArgs.push_back(Args.MakeArgString(llvm::sys::getHostCPUName()));
-    else
-      CmdArgs.push_back(A->getValue());
-  }
 }
 
 void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
@@ -810,7 +802,7 @@ void Flang::ConstructJob(Compilation &C, const JobAction &JA,
   case CodeGenOptions::FramePointerKind::None:
     FPKeepKindStr = "-mframe-pointer=none";
     break;
-  case CodeGenOptions::FramePointerKind::Reserved:
+   case CodeGenOptions::FramePointerKind::Reserved:
     FPKeepKindStr = "-mframe-pointer=reserved";
     break;
   case CodeGenOptions::FramePointerKind::NonLeaf:

clang/test/Preprocessor/embed_weird.cpp

@@ -1,7 +1,9 @@
-// RUN: printf "\0" > %S/Inputs/null_byte.bin
-// RUN: %clang_cc1 %s -fsyntax-only --embed-dir=%S/Inputs -verify=expected,cxx -Wno-c23-extensions
-// RUN: %clang_cc1 -x c -std=c23 %s -fsyntax-only --embed-dir=%S/Inputs -verify=expected,c
-// RUN: rm %S/Inputs/null_byte.bin
+// RUN: rm -rf %t && mkdir -p %t/media
+// RUN: cp %S/Inputs/single_byte.txt %S/Inputs/jk.txt %S/Inputs/numbers.txt %t/
+// RUN: cp %S/Inputs/media/empty %t/media/
+// RUN: printf "\0" > %t/null_byte.bin
+// RUN: %clang_cc1 %s -fsyntax-only --embed-dir=%t -verify=expected,cxx -Wno-c23-extensions
+// RUN: %clang_cc1 -x c -std=c23 %s -fsyntax-only --embed-dir=%t -verify=expected,c
 #embed <media/empty>
 ;
 

compiler-rt/lib/asan/asan_globals.cpp

@@ -344,8 +344,8 @@ void __asan_unregister_image_globals(uptr *flag) {
 }
 
 void __asan_register_elf_globals(uptr *flag, void *start, void *stop) {
-  if (*flag) return;
-  if (!start) return;
+  if (*flag || start == stop)
+    return;
   CHECK_EQ(0, ((uptr)stop - (uptr)start) % sizeof(__asan_global));
   __asan_global *globals_start = (__asan_global*)start;
   __asan_global *globals_stop = (__asan_global*)stop;
@@ -354,8 +354,8 @@ void __asan_register_elf_globals(uptr *flag, void *start, void *stop) {
 }
 
 void __asan_unregister_elf_globals(uptr *flag, void *start, void *stop) {
-  if (!*flag) return;
-  if (!start) return;
+  if (!*flag || start == stop)
+    return;
   CHECK_EQ(0, ((uptr)stop - (uptr)start) % sizeof(__asan_global));
   __asan_global *globals_start = (__asan_global*)start;
   __asan_global *globals_stop = (__asan_global*)stop;

flang/include/flang/Frontend/TargetOptions.h

@@ -32,9 +32,6 @@ class TargetOptions {
   /// If given, the name of the target CPU to generate code for.
   std::string cpu;
 
-  /// If given, the name of the target CPU to tune code for.
-  std::string cpuToTuneFor;
-
   /// The list of target specific features to enable or disable, as written on
   /// the command line.
   std::vector<std::string> featuresAsWritten;

flang/include/flang/Lower/Bridge.h

@@ -65,11 +65,11 @@ class LoweringBridge {
          const Fortran::lower::LoweringOptions &loweringOptions,
          const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults,
          const Fortran::common::LanguageFeatureControl &languageFeatures,
-         const llvm::TargetMachine &targetMachine, llvm::StringRef tuneCPU) {
+         const llvm::TargetMachine &targetMachine) {
     return LoweringBridge(ctx, semanticsContext, defaultKinds, intrinsics,
                           targetCharacteristics, allCooked, triple, kindMap,
                           loweringOptions, envDefaults, languageFeatures,
-                          targetMachine, tuneCPU);
+                          targetMachine);
   }
 
   //===--------------------------------------------------------------------===//
@@ -148,7 +148,7 @@ class LoweringBridge {
       const Fortran::lower::LoweringOptions &loweringOptions,
       const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults,
       const Fortran::common::LanguageFeatureControl &languageFeatures,
-      const llvm::TargetMachine &targetMachine, const llvm::StringRef tuneCPU);
+      const llvm::TargetMachine &targetMachine);
   LoweringBridge() = delete;
   LoweringBridge(const LoweringBridge &) = delete;
 

flang/include/flang/Optimizer/CodeGen/CGPasses.td

@@ -31,8 +31,6 @@ def FIRToLLVMLowering : Pass<"fir-to-llvm-ir", "mlir::ModuleOp"> {
            "Override module's data layout.">,
     Option<"forcedTargetCPU", "target-cpu", "std::string", /*default=*/"",
            "Override module's target CPU.">,
-    Option<"forcedTuneCPU", "tune-cpu", "std::string", /*default=*/"",
-           "Override module's tune CPU.">,
     Option<"forcedTargetFeatures", "target-features", "std::string",
            /*default=*/"", "Override module's target features.">,
     Option<"applyTBAA", "apply-tbaa", "bool", /*default=*/"false",
@@ -70,8 +68,6 @@ def TargetRewritePass : Pass<"target-rewrite", "mlir::ModuleOp"> {
            "Override module's target triple.">,
     Option<"forcedTargetCPU", "target-cpu", "std::string", /*default=*/"",
            "Override module's target CPU.">,
-    Option<"forcedTuneCPU", "tune-cpu", "std::string", /*default=*/"",
-           "Override module's tune CPU.">,
     Option<"forcedTargetFeatures", "target-features", "std::string",
            /*default=*/"", "Override module's target features.">,
     Option<"noCharacterConversion", "no-character-conversion",

flang/include/flang/Optimizer/CodeGen/Target.h

@@ -76,29 +76,14 @@ class CodeGenSpecifics {
       llvm::StringRef targetCPU, mlir::LLVM::TargetFeaturesAttr targetFeatures,
       const mlir::DataLayout &dl);
 
-  static std::unique_ptr<CodeGenSpecifics>
-  get(mlir::MLIRContext *ctx, llvm::Triple &&trp, KindMapping &&kindMap,
-      llvm::StringRef targetCPU, mlir::LLVM::TargetFeaturesAttr targetFeatures,
-      const mlir::DataLayout &dl, llvm::StringRef tuneCPU);
-
   static TypeAndAttr getTypeAndAttr(mlir::Type t) { return TypeAndAttr{t, {}}; }
 
   CodeGenSpecifics(mlir::MLIRContext *ctx, llvm::Triple &&trp,
                    KindMapping &&kindMap, llvm::StringRef targetCPU,
                    mlir::LLVM::TargetFeaturesAttr targetFeatures,
                    const mlir::DataLayout &dl)
       : context{*ctx}, triple{std::move(trp)}, kindMap{std::move(kindMap)},
-        targetCPU{targetCPU}, targetFeatures{targetFeatures}, dataLayout{&dl},
-        tuneCPU{""} {}
-
-  CodeGenSpecifics(mlir::MLIRContext *ctx, llvm::Triple &&trp,
-                   KindMapping &&kindMap, llvm::StringRef targetCPU,
-                   mlir::LLVM::TargetFeaturesAttr targetFeatures,
-                   const mlir::DataLayout &dl, llvm::StringRef tuneCPU)
-      : context{*ctx}, triple{std::move(trp)}, kindMap{std::move(kindMap)},
-        targetCPU{targetCPU}, targetFeatures{targetFeatures}, dataLayout{&dl},
-        tuneCPU{tuneCPU} {}
-
+        targetCPU{targetCPU}, targetFeatures{targetFeatures}, dataLayout{&dl} {}
   CodeGenSpecifics() = delete;
   virtual ~CodeGenSpecifics() {}
 
@@ -180,7 +165,6 @@ class CodeGenSpecifics {
   virtual unsigned char getCIntTypeWidth() const = 0;
 
   llvm::StringRef getTargetCPU() const { return targetCPU; }
-  llvm::StringRef getTuneCPU() const { return tuneCPU; }
 
   mlir::LLVM::TargetFeaturesAttr getTargetFeatures() const {
     return targetFeatures;
@@ -198,7 +182,6 @@ class CodeGenSpecifics {
   llvm::StringRef targetCPU;
   mlir::LLVM::TargetFeaturesAttr targetFeatures;
   const mlir::DataLayout *dataLayout = nullptr;
-  llvm::StringRef tuneCPU;
 };
 
 } // namespace fir

flang/include/flang/Optimizer/Dialect/Support/FIRContext.h

@@ -58,13 +58,6 @@ void setTargetCPU(mlir::ModuleOp mod, llvm::StringRef cpu);
 /// Get the target CPU string from the Module or return a null reference.
 llvm::StringRef getTargetCPU(mlir::ModuleOp mod);
 
-/// Set the tune CPU for the module. `cpu` must not be deallocated while
-/// module `mod` is still live.
-void setTuneCPU(mlir::ModuleOp mod, llvm::StringRef cpu);
-
-/// Get the tune CPU string from the Module or return a null reference.
-llvm::StringRef getTuneCPU(mlir::ModuleOp mod);
-
 /// Set the target features for the module.
 void setTargetFeatures(mlir::ModuleOp mod, llvm::StringRef features);
 

flang/include/flang/Optimizer/Transforms/Passes.td

@@ -411,9 +411,6 @@ def FunctionAttr : Pass<"function-attr", "mlir::func::FuncOp"> {
     Option<"unsafeFPMath", "unsafe-fp-math",
            "bool", /*default=*/"false",
            "Set the unsafe-fp-math attribute on functions in the module.">,
-    Option<"tuneCPU", "tune-cpu",
-           "llvm::StringRef", /*default=*/"llvm::StringRef{}",
-           "Set the tune-cpu attribute on functions in the module.">,
   ];
 }
 

flang/lib/Frontend/CompilerInvocation.cpp

@@ -407,10 +407,6 @@ static void parseTargetArgs(TargetOptions &opts, llvm::opt::ArgList &args) {
           args.getLastArg(clang::driver::options::OPT_target_cpu))
     opts.cpu = a->getValue();
 
-  if (const llvm::opt::Arg *a =
-          args.getLastArg(clang::driver::options::OPT_tune_cpu))
-    opts.cpuToTuneFor = a->getValue();
-
   for (const llvm::opt::Arg *currentArg :
        args.filtered(clang::driver::options::OPT_target_feature))
     opts.featuresAsWritten.emplace_back(currentArg->getValue());

flang/lib/Frontend/FrontendActions.cpp

@@ -297,8 +297,7 @@ bool CodeGenAction::beginSourceFileAction() {
       ci.getParsing().allCooked(), ci.getInvocation().getTargetOpts().triple,
       kindMap, ci.getInvocation().getLoweringOpts(),
       ci.getInvocation().getFrontendOpts().envDefaults,
-      ci.getInvocation().getFrontendOpts().features, targetMachine,
-      ci.getInvocation().getTargetOpts().cpuToTuneFor);
+      ci.getInvocation().getFrontendOpts().features, targetMachine);
 
   // Fetch module from lb, so we can set
   mlirModule = std::make_unique<mlir::ModuleOp>(lb.getModule());

flang/lib/Lower/Bridge.cpp

@@ -5929,7 +5929,7 @@ Fortran::lower::LoweringBridge::LoweringBridge(
     const Fortran::lower::LoweringOptions &loweringOptions,
     const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults,
     const Fortran::common::LanguageFeatureControl &languageFeatures,
-    const llvm::TargetMachine &targetMachine, const llvm::StringRef tuneCPU)
+    const llvm::TargetMachine &targetMachine)
     : semanticsContext{semanticsContext}, defaultKinds{defaultKinds},
       intrinsics{intrinsics}, targetCharacteristics{targetCharacteristics},
       cooked{&cooked}, context{context}, kindMap{kindMap},
@@ -5986,7 +5986,6 @@ Fortran::lower::LoweringBridge::LoweringBridge(
   fir::setTargetTriple(*module.get(), triple);
   fir::setKindMapping(*module.get(), kindMap);
   fir::setTargetCPU(*module.get(), targetMachine.getTargetCPU());
-  fir::setTuneCPU(*module.get(), tuneCPU);
   fir::setTargetFeatures(*module.get(), targetMachine.getTargetFeatureString());
   fir::support::setMLIRDataLayout(*module.get(),
                                   targetMachine.createDataLayout());