[CodeGen][NewPM] Port MachineSink to NPM #115434

optimisan · 2024-11-08T07:20:27Z

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for
the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

optimisan · 2024-11-08T07:20:39Z

[CodeGen][NewPM] Port MachineSink to NPM #115434 👈 (View in Graphite)
[CodeGen][NewPM] Port MachineCycleInfo to NPM #114745 : 2 other dependent PRs (#114746 , #116166 )
[NFC][CodeGen] Clang format MachineSink.cpp #114027
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

llvmbot · 2024-11-08T07:29:24Z

@llvm/pr-subscribers-llvm-analysis
@llvm/pr-subscribers-llvm-ir
@llvm/pr-subscribers-debuginfo
@llvm/pr-subscribers-backend-powerpc
@llvm/pr-subscribers-backend-aarch64

@llvm/pr-subscribers-backend-amdgpu

Author: Akshat Oke (optimisan)

Changes

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for
the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

Patch is 23.77 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/115434.diff

21 Files Affected:

(added) llvm/include/llvm/CodeGen/MachineSink.h (+26)
(modified) llvm/include/llvm/CodeGen/Passes.h (+1-1)
(modified) llvm/include/llvm/InitializePasses.h (+1-1)
(modified) llvm/include/llvm/Passes/CodeGenPassBuilder.h (+2-1)
(modified) llvm/include/llvm/Passes/MachinePassRegistry.def (+8-1)
(modified) llvm/include/llvm/Target/CGPassBuilderOption.h (+1)
(modified) llvm/lib/CodeGen/CodeGen.cpp (+1-1)
(modified) llvm/lib/CodeGen/MachineSink.cpp (+87-40)
(modified) llvm/lib/CodeGen/TargetPassConfig.cpp (+2-2)
(modified) llvm/lib/Passes/PassBuilder.cpp (+6)
(modified) llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp (+1-1)
(modified) llvm/test/CodeGen/AArch64/loop-sink.mir (+1)
(modified) llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir (+2)
(modified) llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir (+2)
(modified) llvm/test/CodeGen/ARM/machine-sink-multidef.mir (+2)
(modified) llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir (+2)
(modified) llvm/test/CodeGen/PowerPC/sink-down-more-instructions-1.mir (+2)
(modified) llvm/test/CodeGen/RISCV/MachineSink-implicit-x0.mir (+1)
(modified) llvm/test/CodeGen/SystemZ/machinesink-dead-cc.mir (+3)
(modified) llvm/test/CodeGen/X86/machinesink-debug-inv-0.mir (+3)
(modified) llvm/test/DebugInfo/MIR/X86/sink-leaves-undef.mir (+1)

diff --git a/llvm/include/llvm/CodeGen/MachineSink.h b/llvm/include/llvm/CodeGen/MachineSink.h
new file mode 100644
index 00000000000000..b5bb59619c0474
--- /dev/null
+++ b/llvm/include/llvm/CodeGen/MachineSink.h
@@ -0,0 +1,26 @@
+//===- MachineSink.h --------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_MACHINESINK_H
+#define LLVM_CODEGEN_MACHINESINK_H
+
+#include "llvm/CodeGen/MachinePassManager.h"
+
+namespace llvm {
+
+class MachineSinkingPass : public PassInfoMixin<MachineSinkingPass> {
+  bool EnableSinkAndFold;
+
+public:
+  MachineSinkingPass(bool EnableSinkAndFold = false)
+      : EnableSinkAndFold(EnableSinkAndFold) {}
+  PreservedAnalyses run(MachineFunction &MF, MachineFunctionAnalysisManager &);
+};
+
+} // namespace llvm
+#endif // LLVM_CODEGEN_MACHINESINK_H
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index 72054ab1d3c21a..708abc86530516 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -349,7 +349,7 @@ namespace llvm {
   extern char &EarlyMachineLICMID;
 
   /// MachineSinking - This pass performs sinking on machine instructions.
-  extern char &MachineSinkingID;
+  extern char &MachineSinkingLegacyID;
 
   /// MachineCopyPropagation - This pass performs copy propagation on
   /// machine instructions.
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 598498f8597b6a..66bb0d10ed4bc6 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -208,7 +208,7 @@ void initializeMachinePostDominatorTreeWrapperPassPass(PassRegistry &);
 void initializeMachineRegionInfoPassPass(PassRegistry &);
 void initializeMachineSanitizerBinaryMetadataPass(PassRegistry &);
 void initializeMachineSchedulerPass(PassRegistry &);
-void initializeMachineSinkingPass(PassRegistry &);
+void initializeMachineSinkingLegacyPass(PassRegistry &);
 void initializeMachineTraceMetricsWrapperPassPass(PassRegistry &);
 void initializeMachineUniformityInfoPrinterPassPass(PassRegistry &);
 void initializeMachineUniformityAnalysisPassPass(PassRegistry &);
diff --git a/llvm/include/llvm/Passes/CodeGenPassBuilder.h b/llvm/include/llvm/Passes/CodeGenPassBuilder.h
index 9e95625fd1d881..2332bb7d8fcbcf 100644
--- a/llvm/include/llvm/Passes/CodeGenPassBuilder.h
+++ b/llvm/include/llvm/Passes/CodeGenPassBuilder.h
@@ -48,6 +48,7 @@
 #include "llvm/CodeGen/MachineLICM.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachinePassManager.h"
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/CodeGen/MachineVerifier.h"
 #include "llvm/CodeGen/OptimizePHIs.h"
 #include "llvm/CodeGen/PHIElimination.h"
@@ -1028,7 +1029,7 @@ void CodeGenPassBuilder<Derived, TargetMachineT>::addMachineSSAOptimization(
   addPass(EarlyMachineLICMPass());
   addPass(MachineCSEPass());
 
-  addPass(MachineSinkingPass());
+  addPass(MachineSinkingPass(Opt.EnableSinkAndFold));
 
   addPass(PeepholeOptimizerPass());
   // Clean-up the dead code that may have been generated by peephole
diff --git a/llvm/include/llvm/Passes/MachinePassRegistry.def b/llvm/include/llvm/Passes/MachinePassRegistry.def
index bbd10c14e99491..041fe9c4e706d0 100644
--- a/llvm/include/llvm/Passes/MachinePassRegistry.def
+++ b/llvm/include/llvm/Passes/MachinePassRegistry.def
@@ -171,6 +171,13 @@ MACHINE_FUNCTION_PASS("verify<machine-trace-metrics>", MachineTraceMetricsVerifi
 #define MACHINE_FUNCTION_PASS_WITH_PARAMS(NAME, CLASS, CREATE_PASS, PARSER,    \
                                           PARAMS)
 #endif
+MACHINE_FUNCTION_PASS_WITH_PARAMS(
+    "machine-sink", "MachineSinkingPass",
+    [](bool EnableSinkAndFold) {
+      return MachineSinkingPass(EnableSinkAndFold);
+    },
+    parseMachineSinkingPassOptions, "enable-sink-fold")
+
 MACHINE_FUNCTION_PASS_WITH_PARAMS(
     "regallocfast", "RegAllocFast",
     [](RegAllocFastPassOptions Opts) { return RegAllocFastPass(Opts); },
@@ -178,6 +185,7 @@ MACHINE_FUNCTION_PASS_WITH_PARAMS(
       return parseRegAllocFastPassOptions(*PB, Params);
     },
     "filter=reg-filter;no-clear-vregs")
+
 #undef MACHINE_FUNCTION_PASS_WITH_PARAMS
 
 // After a pass is converted to new pass manager, its entry should be moved from
@@ -233,7 +241,6 @@ DUMMY_MACHINE_FUNCTION_PASS("machine-function-splitter", MachineFunctionSplitter
 DUMMY_MACHINE_FUNCTION_PASS("machine-latecleanup", MachineLateInstrsCleanupPass)
 DUMMY_MACHINE_FUNCTION_PASS("machine-sanmd", MachineSanitizerBinaryMetadata)
 DUMMY_MACHINE_FUNCTION_PASS("machine-scheduler", MachineSchedulerPass)
-DUMMY_MACHINE_FUNCTION_PASS("machine-sink", MachineSinkingPass)
 DUMMY_MACHINE_FUNCTION_PASS("machine-uniformity", MachineUniformityInfoWrapperPass)
 DUMMY_MACHINE_FUNCTION_PASS("machineinstr-printer", MachineFunctionPrinterPass)
 DUMMY_MACHINE_FUNCTION_PASS("mirfs-discriminators", MIRAddFSDiscriminatorsPass)
diff --git a/llvm/include/llvm/Target/CGPassBuilderOption.h b/llvm/include/llvm/Target/CGPassBuilderOption.h
index 8ab6d63a00056a..77f81902388179 100644
--- a/llvm/include/llvm/Target/CGPassBuilderOption.h
+++ b/llvm/include/llvm/Target/CGPassBuilderOption.h
@@ -32,6 +32,7 @@ struct CGPassBuilderOption {
   bool EnableImplicitNullChecks = false;
   bool EnableBlockPlacementStats = false;
   bool EnableMachineFunctionSplitter = false;
+  bool EnableSinkAndFold = false;
   bool MISchedPostRA = false;
   bool EarlyLiveIntervals = false;
   bool GCEmptyBlocks = false;
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index adddb8daaa0e91..b6881c339ff15e 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -94,7 +94,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeMachinePostDominatorTreeWrapperPassPass(Registry);
   initializeMachineRegionInfoPassPass(Registry);
   initializeMachineSchedulerPass(Registry);
-  initializeMachineSinkingPass(Registry);
+  initializeMachineSinkingLegacyPass(Registry);
   initializeMachineUniformityAnalysisPassPass(Registry);
   initializeMachineUniformityInfoPrinterPassPass(Registry);
   initializeMachineVerifierLegacyPassPass(Registry);
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index a0e09398602e9e..c06cc82cd569ac 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -15,6 +15,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/DepthFirstIterator.h"
 #include "llvm/ADT/MapVector.h"
@@ -36,6 +37,7 @@
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/MachinePassManager.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterClassInfo.h"
@@ -44,6 +46,7 @@
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/IR/Analysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/LLVMContext.h"
@@ -115,7 +118,7 @@ STATISTIC(NumPostRACopySink, "Number of copies sunk after RA");
 
 namespace {
 
-class MachineSinking : public MachineFunctionPass {
+class MachineSinking {
   const TargetSubtargetInfo *STI = nullptr;
   const TargetInstrInfo *TII = nullptr;
   const TargetRegisterInfo *TRI = nullptr;
@@ -127,6 +130,8 @@ class MachineSinking : public MachineFunctionPass {
   const MachineBranchProbabilityInfo *MBPI = nullptr;
   AliasAnalysis *AA = nullptr;
   RegisterClassInfo RegClassInfo;
+  Pass *LegacyPass;
+  MachineFunctionAnalysisManager *MFAM;
 
   // Remember which edges have been considered for breaking.
   SmallSet<std::pair<MachineBasicBlock *, MachineBasicBlock *>, 8>
@@ -182,29 +187,14 @@ class MachineSinking : public MachineFunctionPass {
   bool EnableSinkAndFold;
 
 public:
-  static char ID; // Pass identification
+  MachineSinking(Pass *LegacyPass, MachineFunctionAnalysisManager *MFAM,
+                 bool EnableSinkAndFold)
+      : LegacyPass(LegacyPass), MFAM(MFAM),
+        EnableSinkAndFold(EnableSinkAndFold) {}
 
-  MachineSinking() : MachineFunctionPass(ID) {
-    initializeMachineSinkingPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnMachineFunction(MachineFunction &MF) override;
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    MachineFunctionPass::getAnalysisUsage(AU);
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addRequired<MachineDominatorTreeWrapperPass>();
-    AU.addRequired<MachinePostDominatorTreeWrapperPass>();
-    AU.addRequired<MachineCycleInfoWrapperPass>();
-    AU.addRequired<MachineBranchProbabilityInfoWrapperPass>();
-    AU.addPreserved<MachineCycleInfoWrapperPass>();
-    AU.addPreserved<MachineLoopInfoWrapperPass>();
-    if (UseBlockFreqInfo)
-      AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
-    AU.addRequired<TargetPassConfig>();
-  }
+  bool run(MachineFunction &MF);
 
-  void releaseMemory() override {
+  void releaseMemory() {
     CEBCandidates.clear();
     CEMergeCandidates.clear();
   }
@@ -276,20 +266,45 @@ class MachineSinking : public MachineFunctionPass {
                                        const MachineBasicBlock &MBB);
 };
 
+class MachineSinkingLegacy : public MachineFunctionPass {
+public:
+  static char ID;
+
+  MachineSinkingLegacy() : MachineFunctionPass(ID) {
+    initializeMachineSinkingLegacyPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    MachineFunctionPass::getAnalysisUsage(AU);
+    AU.addRequired<AAResultsWrapperPass>();
+    AU.addRequired<MachineDominatorTreeWrapperPass>();
+    AU.addRequired<MachinePostDominatorTreeWrapperPass>();
+    AU.addRequired<MachineCycleInfoWrapperPass>();
+    AU.addRequired<MachineBranchProbabilityInfoWrapperPass>();
+    AU.addPreserved<MachineCycleInfoWrapperPass>();
+    AU.addPreserved<MachineLoopInfoWrapperPass>();
+    if (UseBlockFreqInfo)
+      AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
+    AU.addRequired<TargetPassConfig>();
+  }
+};
+
 } // end anonymous namespace
 
-char MachineSinking::ID = 0;
+char MachineSinkingLegacy::ID = 0;
 
-char &llvm::MachineSinkingID = MachineSinking::ID;
+char &llvm::MachineSinkingLegacyID = MachineSinkingLegacy::ID;
 
-INITIALIZE_PASS_BEGIN(MachineSinking, DEBUG_TYPE, "Machine code sinking", false,
-                      false)
+INITIALIZE_PASS_BEGIN(MachineSinkingLegacy, DEBUG_TYPE, "Machine code sinking",
+                      false, false)
 INITIALIZE_PASS_DEPENDENCY(MachineBranchProbabilityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(MachineDominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(MachineCycleInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_END(MachineSinking, DEBUG_TYPE, "Machine code sinking", false,
-                    false)
+INITIALIZE_PASS_END(MachineSinkingLegacy, DEBUG_TYPE, "Machine code sinking",
+                    false, false)
 
 /// Return true if a target defined block prologue instruction interferes
 /// with a sink candidate.
@@ -709,27 +724,57 @@ void MachineSinking::FindCycleSinkCandidates(
   }
 }
 
-bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
+PreservedAnalyses
+MachineSinkingPass::run(MachineFunction &MF,
+                        MachineFunctionAnalysisManager &MFAM) {
+  if (MF.getFunction().hasOptNone())
+    return PreservedAnalyses::all();
+
+  MachineSinking Impl(nullptr, &MFAM, EnableSinkAndFold);
+  bool Changed = Impl.run(MF);
+  if (!Changed)
+    return PreservedAnalyses::all();
+  auto PA = getMachineFunctionPassPreservedAnalyses();
+  PA.preserve<MachineCycleAnalysis>();
+  PA.preserve<MachineLoopAnalysis>();
+  return PA;
+}
+
+bool MachineSinkingLegacy::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
+  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
+  bool EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
+
+  MachineSinking Impl(this, nullptr, EnableSinkAndFold);
+  return Impl.run(MF);
+}
+
+#define GET_ANALYSIS(ANALYSIS, INFIX, GETTER)                                  \
+  ((LegacyPass)                                                                \
+       ? &LegacyPass->getAnalysis<ANALYSIS##INFIX##WrapperPass>().GETTER()     \
+       : &MFAM->getResult<ANALYSIS##Analysis>(MF))
+
+bool MachineSinking::run(MachineFunction &MF) {
   LLVM_DEBUG(dbgs() << "******** Machine Sinking ********\n");
 
   STI = &MF.getSubtarget();
   TII = STI->getInstrInfo();
   TRI = STI->getRegisterInfo();
   MRI = &MF.getRegInfo();
-  DT = &getAnalysis<MachineDominatorTreeWrapperPass>().getDomTree();
-  PDT = &getAnalysis<MachinePostDominatorTreeWrapperPass>().getPostDomTree();
-  CI = &getAnalysis<MachineCycleInfoWrapperPass>().getCycleInfo();
-  MBFI = UseBlockFreqInfo
-             ? &getAnalysis<MachineBlockFrequencyInfoWrapperPass>().getMBFI()
-             : nullptr;
-  MBPI = &getAnalysis<MachineBranchProbabilityInfoWrapperPass>().getMBPI();
-  AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+  DT = GET_ANALYSIS(MachineDominatorTree, , getDomTree);
+  PDT = GET_ANALYSIS(MachinePostDominatorTree, , getPostDomTree);
+  CI = GET_ANALYSIS(MachineCycle, Info, getCycleInfo);
+  MBFI = UseBlockFreqInfo ? GET_ANALYSIS(MachineBlockFrequency, Info, getMBFI)
+                          : nullptr;
+  MBPI = GET_ANALYSIS(MachineBranchProbability, Info, getMBPI);
+  AA = (LegacyPass)
+           ? &LegacyPass->getAnalysis<AAResultsWrapperPass>().getAAResults()
+           : &MFAM->getResult<FunctionAnalysisManagerMachineFunctionProxy>(MF)
+                  .getManager()
+                  .getResult<AAManager>(MF.getFunction());
   RegClassInfo.runOnMachineFunction(MF);
-  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
-  EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
 
   bool EverMadeChange = false;
 
@@ -745,7 +790,8 @@ bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
 
     // If we have anything we marked as toSplit, split it now.
     for (const auto &Pair : ToSplit) {
-      auto NewSucc = Pair.first->SplitCriticalEdge(Pair.second, *this);
+      auto NewSucc =
+          Pair.first->SplitCriticalEdge(Pair.second, LegacyPass, MFAM, nullptr);
       if (NewSucc != nullptr) {
         LLVM_DEBUG(dbgs() << " *** Splitting critical edge: "
                           << printMBBReference(*Pair.first) << " -- "
@@ -804,6 +850,7 @@ bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
     MRI->clearKillFlags(I);
   RegsToClearKillFlags.clear();
 
+  releaseMemory();
   return EverMadeChange;
 }
 
diff --git a/llvm/lib/CodeGen/TargetPassConfig.cpp b/llvm/lib/CodeGen/TargetPassConfig.cpp
index 782ed6062191ce..be71488e39458d 100644
--- a/llvm/lib/CodeGen/TargetPassConfig.cpp
+++ b/llvm/lib/CodeGen/TargetPassConfig.cpp
@@ -317,7 +317,7 @@ static IdentifyingPassPtr overridePass(AnalysisID StandardID,
   if (StandardID == &MachineLICMID)
     return applyDisable(TargetID, DisablePostRAMachineLICM);
 
-  if (StandardID == &MachineSinkingID)
+  if (StandardID == &MachineSinkingLegacyID)
     return applyDisable(TargetID, DisableMachineSink);
 
   if (StandardID == &PostRAMachineSinkingID)
@@ -1304,7 +1304,7 @@ void TargetPassConfig::addMachineSSAOptimization() {
   addPass(&EarlyMachineLICMID);
   addPass(&MachineCSELegacyID);
 
-  addPass(&MachineSinkingID);
+  addPass(&MachineSinkingLegacyID);
 
   addPass(&PeepholeOptimizerID);
   // Clean-up the dead code that may have been generated by peephole
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index bd670c0b13df1d..b7a2c881d495c3 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -114,6 +114,7 @@
 #include "llvm/CodeGen/MachinePassManager.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/CodeGen/MachineTraceMetrics.h"
 #include "llvm/CodeGen/MachineVerifier.h"
 #include "llvm/CodeGen/OptimizePHIs.h"
@@ -1277,6 +1278,11 @@ Expected<RealtimeSanitizerOptions> parseRtSanPassOptions(StringRef Params) {
   return Result;
 }
 
+Expected<bool> parseMachineSinkingPassOptions(StringRef Params) {
+  return PassBuilder::parseSinglePassOption(Params, "enable-sink-fold",
+                                            "MachineSinkingPass");
+}
+
 } // namespace
 
 /// Tests whether a pass name starts with a valid prefix for a default pipeline
diff --git a/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp b/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
index a5a147da8da1c5..20c86362ded2a6 100644
--- a/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
@@ -492,7 +492,7 @@ void NVPTXPassConfig::addMachineSSAOptimization() {
   addPass(&EarlyMachineLICMID);
   addPass(&MachineCSELegacyID);
 
-  addPass(&MachineSinkingID);
+  addPass(&MachineSinkingLegacyID);
   printAndVerify("After Machine LICM, CSE and Sinking passes");
 
   addPass(&PeepholeOptimizerID);
diff --git a/llvm/test/CodeGen/AArch64/loop-sink.mir b/llvm/test/CodeGen/AArch64/loop-sink.mir
index 36d39ffbadc29a..de81d99369394b 100644
--- a/llvm/test/CodeGen/AArch64/loop-sink.mir
+++ b/llvm/test/CodeGen/AArch64/loop-sink.mir
@@ -1,5 +1,6 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple aarch64 -run-pass=machine-sink  -sink-insts-to-avoid-spills -aarch64-enable-sink-fold=true %s -o - 2>&1 | FileCheck %s
+# RUN: llc -mtriple aarch64 -passes="machine-sink<enable-sink-fold>" -sink-insts-to-avoid-spills %s -o - 2>&1 | FileCheck %s
 --- |
   target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64"
diff --git a/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir b/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
index be5737d297b290..f146c4436bbfa8 100644
--- a/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
+++ b/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
@@ -1,4 +1,6 @@
 # RUN: llc %s -run-pass=machine-sink --aarch64-enable-sink-fold=true -o - | FileCheck %s
+
+# RUN: llc %s -passes="machine-sink<enable-sink-fold>" -o - | FileCheck %s
 --- |
   target triple = "aarch64-linux"
 
diff --git a/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir b/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
index 329f2967121603..7ef6d37ddc1832 100644
--- a/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
+++ b/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
@@ -1,6 +1,8 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 3
 # RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1031 -run-pass=machine-sink -o -  %s | FileCheck %s
 
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1031 -passes=machine-sink -o -  %s | FileCheck %s
+
 ---
 name: machine-sink-temporal-divergence
 tracksRegLiveness: true
diff --git a/llvm/test/CodeGen/ARM/machine-sink-multidef.mir b/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
index 737ca824265d6b..485e659a53ab6b 100644
--- a/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
+++ b/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
@@ -1,6 +1,8 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc %s -o - -run-pass=machine-sink -mtriple=arm-none-eabi | FileCheck %s
 
+# RUN: llc %s -o - -passes=machine-sink -mtriple=arm-none-eabi | FileCheck %s
+
 --- |
   target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
   target triple = "arm-none-unknown-eabi"
diff --git a/llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir b/llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir
i...
[truncated]

llvmbot · 2024-11-08T07:29:24Z

@llvm/pr-subscribers-backend-hexagon

Author: Akshat Oke (optimisan)

Changes

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for
the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

Patch is 23.77 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/115434.diff

21 Files Affected:

(added) llvm/include/llvm/CodeGen/MachineSink.h (+26)
(modified) llvm/include/llvm/CodeGen/Passes.h (+1-1)
(modified) llvm/include/llvm/InitializePasses.h (+1-1)
(modified) llvm/include/llvm/Passes/CodeGenPassBuilder.h (+2-1)
(modified) llvm/include/llvm/Passes/MachinePassRegistry.def (+8-1)
(modified) llvm/include/llvm/Target/CGPassBuilderOption.h (+1)
(modified) llvm/lib/CodeGen/CodeGen.cpp (+1-1)
(modified) llvm/lib/CodeGen/MachineSink.cpp (+87-40)
(modified) llvm/lib/CodeGen/TargetPassConfig.cpp (+2-2)
(modified) llvm/lib/Passes/PassBuilder.cpp (+6)
(modified) llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp (+1-1)
(modified) llvm/test/CodeGen/AArch64/loop-sink.mir (+1)
(modified) llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir (+2)
(modified) llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir (+2)
(modified) llvm/test/CodeGen/ARM/machine-sink-multidef.mir (+2)
(modified) llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir (+2)
(modified) llvm/test/CodeGen/PowerPC/sink-down-more-instructions-1.mir (+2)
(modified) llvm/test/CodeGen/RISCV/MachineSink-implicit-x0.mir (+1)
(modified) llvm/test/CodeGen/SystemZ/machinesink-dead-cc.mir (+3)
(modified) llvm/test/CodeGen/X86/machinesink-debug-inv-0.mir (+3)
(modified) llvm/test/DebugInfo/MIR/X86/sink-leaves-undef.mir (+1)

diff --git a/llvm/include/llvm/CodeGen/MachineSink.h b/llvm/include/llvm/CodeGen/MachineSink.h
new file mode 100644
index 00000000000000..b5bb59619c0474
--- /dev/null
+++ b/llvm/include/llvm/CodeGen/MachineSink.h
@@ -0,0 +1,26 @@
+//===- MachineSink.h --------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_MACHINESINK_H
+#define LLVM_CODEGEN_MACHINESINK_H
+
+#include "llvm/CodeGen/MachinePassManager.h"
+
+namespace llvm {
+
+class MachineSinkingPass : public PassInfoMixin<MachineSinkingPass> {
+  bool EnableSinkAndFold;
+
+public:
+  MachineSinkingPass(bool EnableSinkAndFold = false)
+      : EnableSinkAndFold(EnableSinkAndFold) {}
+  PreservedAnalyses run(MachineFunction &MF, MachineFunctionAnalysisManager &);
+};
+
+} // namespace llvm
+#endif // LLVM_CODEGEN_MACHINESINK_H
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index 72054ab1d3c21a..708abc86530516 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -349,7 +349,7 @@ namespace llvm {
   extern char &EarlyMachineLICMID;
 
   /// MachineSinking - This pass performs sinking on machine instructions.
-  extern char &MachineSinkingID;
+  extern char &MachineSinkingLegacyID;
 
   /// MachineCopyPropagation - This pass performs copy propagation on
   /// machine instructions.
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 598498f8597b6a..66bb0d10ed4bc6 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -208,7 +208,7 @@ void initializeMachinePostDominatorTreeWrapperPassPass(PassRegistry &);
 void initializeMachineRegionInfoPassPass(PassRegistry &);
 void initializeMachineSanitizerBinaryMetadataPass(PassRegistry &);
 void initializeMachineSchedulerPass(PassRegistry &);
-void initializeMachineSinkingPass(PassRegistry &);
+void initializeMachineSinkingLegacyPass(PassRegistry &);
 void initializeMachineTraceMetricsWrapperPassPass(PassRegistry &);
 void initializeMachineUniformityInfoPrinterPassPass(PassRegistry &);
 void initializeMachineUniformityAnalysisPassPass(PassRegistry &);
diff --git a/llvm/include/llvm/Passes/CodeGenPassBuilder.h b/llvm/include/llvm/Passes/CodeGenPassBuilder.h
index 9e95625fd1d881..2332bb7d8fcbcf 100644
--- a/llvm/include/llvm/Passes/CodeGenPassBuilder.h
+++ b/llvm/include/llvm/Passes/CodeGenPassBuilder.h
@@ -48,6 +48,7 @@
 #include "llvm/CodeGen/MachineLICM.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachinePassManager.h"
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/CodeGen/MachineVerifier.h"
 #include "llvm/CodeGen/OptimizePHIs.h"
 #include "llvm/CodeGen/PHIElimination.h"
@@ -1028,7 +1029,7 @@ void CodeGenPassBuilder<Derived, TargetMachineT>::addMachineSSAOptimization(
   addPass(EarlyMachineLICMPass());
   addPass(MachineCSEPass());
 
-  addPass(MachineSinkingPass());
+  addPass(MachineSinkingPass(Opt.EnableSinkAndFold));
 
   addPass(PeepholeOptimizerPass());
   // Clean-up the dead code that may have been generated by peephole
diff --git a/llvm/include/llvm/Passes/MachinePassRegistry.def b/llvm/include/llvm/Passes/MachinePassRegistry.def
index bbd10c14e99491..041fe9c4e706d0 100644
--- a/llvm/include/llvm/Passes/MachinePassRegistry.def
+++ b/llvm/include/llvm/Passes/MachinePassRegistry.def
@@ -171,6 +171,13 @@ MACHINE_FUNCTION_PASS("verify<machine-trace-metrics>", MachineTraceMetricsVerifi
 #define MACHINE_FUNCTION_PASS_WITH_PARAMS(NAME, CLASS, CREATE_PASS, PARSER,    \
                                           PARAMS)
 #endif
+MACHINE_FUNCTION_PASS_WITH_PARAMS(
+    "machine-sink", "MachineSinkingPass",
+    [](bool EnableSinkAndFold) {
+      return MachineSinkingPass(EnableSinkAndFold);
+    },
+    parseMachineSinkingPassOptions, "enable-sink-fold")
+
 MACHINE_FUNCTION_PASS_WITH_PARAMS(
     "regallocfast", "RegAllocFast",
     [](RegAllocFastPassOptions Opts) { return RegAllocFastPass(Opts); },
@@ -178,6 +185,7 @@ MACHINE_FUNCTION_PASS_WITH_PARAMS(
       return parseRegAllocFastPassOptions(*PB, Params);
     },
     "filter=reg-filter;no-clear-vregs")
+
 #undef MACHINE_FUNCTION_PASS_WITH_PARAMS
 
 // After a pass is converted to new pass manager, its entry should be moved from
@@ -233,7 +241,6 @@ DUMMY_MACHINE_FUNCTION_PASS("machine-function-splitter", MachineFunctionSplitter
 DUMMY_MACHINE_FUNCTION_PASS("machine-latecleanup", MachineLateInstrsCleanupPass)
 DUMMY_MACHINE_FUNCTION_PASS("machine-sanmd", MachineSanitizerBinaryMetadata)
 DUMMY_MACHINE_FUNCTION_PASS("machine-scheduler", MachineSchedulerPass)
-DUMMY_MACHINE_FUNCTION_PASS("machine-sink", MachineSinkingPass)
 DUMMY_MACHINE_FUNCTION_PASS("machine-uniformity", MachineUniformityInfoWrapperPass)
 DUMMY_MACHINE_FUNCTION_PASS("machineinstr-printer", MachineFunctionPrinterPass)
 DUMMY_MACHINE_FUNCTION_PASS("mirfs-discriminators", MIRAddFSDiscriminatorsPass)
diff --git a/llvm/include/llvm/Target/CGPassBuilderOption.h b/llvm/include/llvm/Target/CGPassBuilderOption.h
index 8ab6d63a00056a..77f81902388179 100644
--- a/llvm/include/llvm/Target/CGPassBuilderOption.h
+++ b/llvm/include/llvm/Target/CGPassBuilderOption.h
@@ -32,6 +32,7 @@ struct CGPassBuilderOption {
   bool EnableImplicitNullChecks = false;
   bool EnableBlockPlacementStats = false;
   bool EnableMachineFunctionSplitter = false;
+  bool EnableSinkAndFold = false;
   bool MISchedPostRA = false;
   bool EarlyLiveIntervals = false;
   bool GCEmptyBlocks = false;
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index adddb8daaa0e91..b6881c339ff15e 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -94,7 +94,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeMachinePostDominatorTreeWrapperPassPass(Registry);
   initializeMachineRegionInfoPassPass(Registry);
   initializeMachineSchedulerPass(Registry);
-  initializeMachineSinkingPass(Registry);
+  initializeMachineSinkingLegacyPass(Registry);
   initializeMachineUniformityAnalysisPassPass(Registry);
   initializeMachineUniformityInfoPrinterPassPass(Registry);
   initializeMachineVerifierLegacyPassPass(Registry);
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index a0e09398602e9e..c06cc82cd569ac 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -15,6 +15,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/DepthFirstIterator.h"
 #include "llvm/ADT/MapVector.h"
@@ -36,6 +37,7 @@
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/MachinePassManager.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterClassInfo.h"
@@ -44,6 +46,7 @@
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/IR/Analysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/LLVMContext.h"
@@ -115,7 +118,7 @@ STATISTIC(NumPostRACopySink, "Number of copies sunk after RA");
 
 namespace {
 
-class MachineSinking : public MachineFunctionPass {
+class MachineSinking {
   const TargetSubtargetInfo *STI = nullptr;
   const TargetInstrInfo *TII = nullptr;
   const TargetRegisterInfo *TRI = nullptr;
@@ -127,6 +130,8 @@ class MachineSinking : public MachineFunctionPass {
   const MachineBranchProbabilityInfo *MBPI = nullptr;
   AliasAnalysis *AA = nullptr;
   RegisterClassInfo RegClassInfo;
+  Pass *LegacyPass;
+  MachineFunctionAnalysisManager *MFAM;
 
   // Remember which edges have been considered for breaking.
   SmallSet<std::pair<MachineBasicBlock *, MachineBasicBlock *>, 8>
@@ -182,29 +187,14 @@ class MachineSinking : public MachineFunctionPass {
   bool EnableSinkAndFold;
 
 public:
-  static char ID; // Pass identification
+  MachineSinking(Pass *LegacyPass, MachineFunctionAnalysisManager *MFAM,
+                 bool EnableSinkAndFold)
+      : LegacyPass(LegacyPass), MFAM(MFAM),
+        EnableSinkAndFold(EnableSinkAndFold) {}
 
-  MachineSinking() : MachineFunctionPass(ID) {
-    initializeMachineSinkingPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnMachineFunction(MachineFunction &MF) override;
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    MachineFunctionPass::getAnalysisUsage(AU);
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addRequired<MachineDominatorTreeWrapperPass>();
-    AU.addRequired<MachinePostDominatorTreeWrapperPass>();
-    AU.addRequired<MachineCycleInfoWrapperPass>();
-    AU.addRequired<MachineBranchProbabilityInfoWrapperPass>();
-    AU.addPreserved<MachineCycleInfoWrapperPass>();
-    AU.addPreserved<MachineLoopInfoWrapperPass>();
-    if (UseBlockFreqInfo)
-      AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
-    AU.addRequired<TargetPassConfig>();
-  }
+  bool run(MachineFunction &MF);
 
-  void releaseMemory() override {
+  void releaseMemory() {
     CEBCandidates.clear();
     CEMergeCandidates.clear();
   }
@@ -276,20 +266,45 @@ class MachineSinking : public MachineFunctionPass {
                                        const MachineBasicBlock &MBB);
 };
 
+class MachineSinkingLegacy : public MachineFunctionPass {
+public:
+  static char ID;
+
+  MachineSinkingLegacy() : MachineFunctionPass(ID) {
+    initializeMachineSinkingLegacyPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    MachineFunctionPass::getAnalysisUsage(AU);
+    AU.addRequired<AAResultsWrapperPass>();
+    AU.addRequired<MachineDominatorTreeWrapperPass>();
+    AU.addRequired<MachinePostDominatorTreeWrapperPass>();
+    AU.addRequired<MachineCycleInfoWrapperPass>();
+    AU.addRequired<MachineBranchProbabilityInfoWrapperPass>();
+    AU.addPreserved<MachineCycleInfoWrapperPass>();
+    AU.addPreserved<MachineLoopInfoWrapperPass>();
+    if (UseBlockFreqInfo)
+      AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
+    AU.addRequired<TargetPassConfig>();
+  }
+};
+
 } // end anonymous namespace
 
-char MachineSinking::ID = 0;
+char MachineSinkingLegacy::ID = 0;
 
-char &llvm::MachineSinkingID = MachineSinking::ID;
+char &llvm::MachineSinkingLegacyID = MachineSinkingLegacy::ID;
 
-INITIALIZE_PASS_BEGIN(MachineSinking, DEBUG_TYPE, "Machine code sinking", false,
-                      false)
+INITIALIZE_PASS_BEGIN(MachineSinkingLegacy, DEBUG_TYPE, "Machine code sinking",
+                      false, false)
 INITIALIZE_PASS_DEPENDENCY(MachineBranchProbabilityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(MachineDominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(MachineCycleInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_END(MachineSinking, DEBUG_TYPE, "Machine code sinking", false,
-                    false)
+INITIALIZE_PASS_END(MachineSinkingLegacy, DEBUG_TYPE, "Machine code sinking",
+                    false, false)
 
 /// Return true if a target defined block prologue instruction interferes
 /// with a sink candidate.
@@ -709,27 +724,57 @@ void MachineSinking::FindCycleSinkCandidates(
   }
 }
 
-bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
+PreservedAnalyses
+MachineSinkingPass::run(MachineFunction &MF,
+                        MachineFunctionAnalysisManager &MFAM) {
+  if (MF.getFunction().hasOptNone())
+    return PreservedAnalyses::all();
+
+  MachineSinking Impl(nullptr, &MFAM, EnableSinkAndFold);
+  bool Changed = Impl.run(MF);
+  if (!Changed)
+    return PreservedAnalyses::all();
+  auto PA = getMachineFunctionPassPreservedAnalyses();
+  PA.preserve<MachineCycleAnalysis>();
+  PA.preserve<MachineLoopAnalysis>();
+  return PA;
+}
+
+bool MachineSinkingLegacy::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
+  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
+  bool EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
+
+  MachineSinking Impl(this, nullptr, EnableSinkAndFold);
+  return Impl.run(MF);
+}
+
+#define GET_ANALYSIS(ANALYSIS, INFIX, GETTER)                                  \
+  ((LegacyPass)                                                                \
+       ? &LegacyPass->getAnalysis<ANALYSIS##INFIX##WrapperPass>().GETTER()     \
+       : &MFAM->getResult<ANALYSIS##Analysis>(MF))
+
+bool MachineSinking::run(MachineFunction &MF) {
   LLVM_DEBUG(dbgs() << "******** Machine Sinking ********\n");
 
   STI = &MF.getSubtarget();
   TII = STI->getInstrInfo();
   TRI = STI->getRegisterInfo();
   MRI = &MF.getRegInfo();
-  DT = &getAnalysis<MachineDominatorTreeWrapperPass>().getDomTree();
-  PDT = &getAnalysis<MachinePostDominatorTreeWrapperPass>().getPostDomTree();
-  CI = &getAnalysis<MachineCycleInfoWrapperPass>().getCycleInfo();
-  MBFI = UseBlockFreqInfo
-             ? &getAnalysis<MachineBlockFrequencyInfoWrapperPass>().getMBFI()
-             : nullptr;
-  MBPI = &getAnalysis<MachineBranchProbabilityInfoWrapperPass>().getMBPI();
-  AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+  DT = GET_ANALYSIS(MachineDominatorTree, , getDomTree);
+  PDT = GET_ANALYSIS(MachinePostDominatorTree, , getPostDomTree);
+  CI = GET_ANALYSIS(MachineCycle, Info, getCycleInfo);
+  MBFI = UseBlockFreqInfo ? GET_ANALYSIS(MachineBlockFrequency, Info, getMBFI)
+                          : nullptr;
+  MBPI = GET_ANALYSIS(MachineBranchProbability, Info, getMBPI);
+  AA = (LegacyPass)
+           ? &LegacyPass->getAnalysis<AAResultsWrapperPass>().getAAResults()
+           : &MFAM->getResult<FunctionAnalysisManagerMachineFunctionProxy>(MF)
+                  .getManager()
+                  .getResult<AAManager>(MF.getFunction());
   RegClassInfo.runOnMachineFunction(MF);
-  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
-  EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
 
   bool EverMadeChange = false;
 
@@ -745,7 +790,8 @@ bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
 
     // If we have anything we marked as toSplit, split it now.
     for (const auto &Pair : ToSplit) {
-      auto NewSucc = Pair.first->SplitCriticalEdge(Pair.second, *this);
+      auto NewSucc =
+          Pair.first->SplitCriticalEdge(Pair.second, LegacyPass, MFAM, nullptr);
       if (NewSucc != nullptr) {
         LLVM_DEBUG(dbgs() << " *** Splitting critical edge: "
                           << printMBBReference(*Pair.first) << " -- "
@@ -804,6 +850,7 @@ bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
     MRI->clearKillFlags(I);
   RegsToClearKillFlags.clear();
 
+  releaseMemory();
   return EverMadeChange;
 }
 
diff --git a/llvm/lib/CodeGen/TargetPassConfig.cpp b/llvm/lib/CodeGen/TargetPassConfig.cpp
index 782ed6062191ce..be71488e39458d 100644
--- a/llvm/lib/CodeGen/TargetPassConfig.cpp
+++ b/llvm/lib/CodeGen/TargetPassConfig.cpp
@@ -317,7 +317,7 @@ static IdentifyingPassPtr overridePass(AnalysisID StandardID,
   if (StandardID == &MachineLICMID)
     return applyDisable(TargetID, DisablePostRAMachineLICM);
 
-  if (StandardID == &MachineSinkingID)
+  if (StandardID == &MachineSinkingLegacyID)
     return applyDisable(TargetID, DisableMachineSink);
 
   if (StandardID == &PostRAMachineSinkingID)
@@ -1304,7 +1304,7 @@ void TargetPassConfig::addMachineSSAOptimization() {
   addPass(&EarlyMachineLICMID);
   addPass(&MachineCSELegacyID);
 
-  addPass(&MachineSinkingID);
+  addPass(&MachineSinkingLegacyID);
 
   addPass(&PeepholeOptimizerID);
   // Clean-up the dead code that may have been generated by peephole
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index bd670c0b13df1d..b7a2c881d495c3 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -114,6 +114,7 @@
 #include "llvm/CodeGen/MachinePassManager.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/CodeGen/MachineTraceMetrics.h"
 #include "llvm/CodeGen/MachineVerifier.h"
 #include "llvm/CodeGen/OptimizePHIs.h"
@@ -1277,6 +1278,11 @@ Expected<RealtimeSanitizerOptions> parseRtSanPassOptions(StringRef Params) {
   return Result;
 }
 
+Expected<bool> parseMachineSinkingPassOptions(StringRef Params) {
+  return PassBuilder::parseSinglePassOption(Params, "enable-sink-fold",
+                                            "MachineSinkingPass");
+}
+
 } // namespace
 
 /// Tests whether a pass name starts with a valid prefix for a default pipeline
diff --git a/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp b/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
index a5a147da8da1c5..20c86362ded2a6 100644
--- a/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
@@ -492,7 +492,7 @@ void NVPTXPassConfig::addMachineSSAOptimization() {
   addPass(&EarlyMachineLICMID);
   addPass(&MachineCSELegacyID);
 
-  addPass(&MachineSinkingID);
+  addPass(&MachineSinkingLegacyID);
   printAndVerify("After Machine LICM, CSE and Sinking passes");
 
   addPass(&PeepholeOptimizerID);
diff --git a/llvm/test/CodeGen/AArch64/loop-sink.mir b/llvm/test/CodeGen/AArch64/loop-sink.mir
index 36d39ffbadc29a..de81d99369394b 100644
--- a/llvm/test/CodeGen/AArch64/loop-sink.mir
+++ b/llvm/test/CodeGen/AArch64/loop-sink.mir
@@ -1,5 +1,6 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple aarch64 -run-pass=machine-sink  -sink-insts-to-avoid-spills -aarch64-enable-sink-fold=true %s -o - 2>&1 | FileCheck %s
+# RUN: llc -mtriple aarch64 -passes="machine-sink<enable-sink-fold>" -sink-insts-to-avoid-spills %s -o - 2>&1 | FileCheck %s
 --- |
   target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64"
diff --git a/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir b/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
index be5737d297b290..f146c4436bbfa8 100644
--- a/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
+++ b/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
@@ -1,4 +1,6 @@
 # RUN: llc %s -run-pass=machine-sink --aarch64-enable-sink-fold=true -o - | FileCheck %s
+
+# RUN: llc %s -passes="machine-sink<enable-sink-fold>" -o - | FileCheck %s
 --- |
   target triple = "aarch64-linux"
 
diff --git a/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir b/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
index 329f2967121603..7ef6d37ddc1832 100644
--- a/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
+++ b/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
@@ -1,6 +1,8 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 3
 # RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1031 -run-pass=machine-sink -o -  %s | FileCheck %s
 
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1031 -passes=machine-sink -o -  %s | FileCheck %s
+
 ---
 name: machine-sink-temporal-divergence
 tracksRegLiveness: true
diff --git a/llvm/test/CodeGen/ARM/machine-sink-multidef.mir b/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
index 737ca824265d6b..485e659a53ab6b 100644
--- a/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
+++ b/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
@@ -1,6 +1,8 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc %s -o - -run-pass=machine-sink -mtriple=arm-none-eabi | FileCheck %s
 
+# RUN: llc %s -o - -passes=machine-sink -mtriple=arm-none-eabi | FileCheck %s
+
 --- |
   target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
   target triple = "arm-none-unknown-eabi"
diff --git a/llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir b/llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir
i...
[truncated]

llvmbot · 2024-11-08T07:29:25Z

@llvm/pr-subscribers-backend-x86

Author: Akshat Oke (optimisan)

Changes

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for
the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

Patch is 23.77 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/115434.diff

21 Files Affected:

(added) llvm/include/llvm/CodeGen/MachineSink.h (+26)
(modified) llvm/include/llvm/CodeGen/Passes.h (+1-1)
(modified) llvm/include/llvm/InitializePasses.h (+1-1)
(modified) llvm/include/llvm/Passes/CodeGenPassBuilder.h (+2-1)
(modified) llvm/include/llvm/Passes/MachinePassRegistry.def (+8-1)
(modified) llvm/include/llvm/Target/CGPassBuilderOption.h (+1)
(modified) llvm/lib/CodeGen/CodeGen.cpp (+1-1)
(modified) llvm/lib/CodeGen/MachineSink.cpp (+87-40)
(modified) llvm/lib/CodeGen/TargetPassConfig.cpp (+2-2)
(modified) llvm/lib/Passes/PassBuilder.cpp (+6)
(modified) llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp (+1-1)
(modified) llvm/test/CodeGen/AArch64/loop-sink.mir (+1)
(modified) llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir (+2)
(modified) llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir (+2)
(modified) llvm/test/CodeGen/ARM/machine-sink-multidef.mir (+2)
(modified) llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir (+2)
(modified) llvm/test/CodeGen/PowerPC/sink-down-more-instructions-1.mir (+2)
(modified) llvm/test/CodeGen/RISCV/MachineSink-implicit-x0.mir (+1)
(modified) llvm/test/CodeGen/SystemZ/machinesink-dead-cc.mir (+3)
(modified) llvm/test/CodeGen/X86/machinesink-debug-inv-0.mir (+3)
(modified) llvm/test/DebugInfo/MIR/X86/sink-leaves-undef.mir (+1)

diff --git a/llvm/include/llvm/CodeGen/MachineSink.h b/llvm/include/llvm/CodeGen/MachineSink.h
new file mode 100644
index 00000000000000..b5bb59619c0474
--- /dev/null
+++ b/llvm/include/llvm/CodeGen/MachineSink.h
@@ -0,0 +1,26 @@
+//===- MachineSink.h --------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_MACHINESINK_H
+#define LLVM_CODEGEN_MACHINESINK_H
+
+#include "llvm/CodeGen/MachinePassManager.h"
+
+namespace llvm {
+
+class MachineSinkingPass : public PassInfoMixin<MachineSinkingPass> {
+  bool EnableSinkAndFold;
+
+public:
+  MachineSinkingPass(bool EnableSinkAndFold = false)
+      : EnableSinkAndFold(EnableSinkAndFold) {}
+  PreservedAnalyses run(MachineFunction &MF, MachineFunctionAnalysisManager &);
+};
+
+} // namespace llvm
+#endif // LLVM_CODEGEN_MACHINESINK_H
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index 72054ab1d3c21a..708abc86530516 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -349,7 +349,7 @@ namespace llvm {
   extern char &EarlyMachineLICMID;
 
   /// MachineSinking - This pass performs sinking on machine instructions.
-  extern char &MachineSinkingID;
+  extern char &MachineSinkingLegacyID;
 
   /// MachineCopyPropagation - This pass performs copy propagation on
   /// machine instructions.
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 598498f8597b6a..66bb0d10ed4bc6 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -208,7 +208,7 @@ void initializeMachinePostDominatorTreeWrapperPassPass(PassRegistry &);
 void initializeMachineRegionInfoPassPass(PassRegistry &);
 void initializeMachineSanitizerBinaryMetadataPass(PassRegistry &);
 void initializeMachineSchedulerPass(PassRegistry &);
-void initializeMachineSinkingPass(PassRegistry &);
+void initializeMachineSinkingLegacyPass(PassRegistry &);
 void initializeMachineTraceMetricsWrapperPassPass(PassRegistry &);
 void initializeMachineUniformityInfoPrinterPassPass(PassRegistry &);
 void initializeMachineUniformityAnalysisPassPass(PassRegistry &);
diff --git a/llvm/include/llvm/Passes/CodeGenPassBuilder.h b/llvm/include/llvm/Passes/CodeGenPassBuilder.h
index 9e95625fd1d881..2332bb7d8fcbcf 100644
--- a/llvm/include/llvm/Passes/CodeGenPassBuilder.h
+++ b/llvm/include/llvm/Passes/CodeGenPassBuilder.h
@@ -48,6 +48,7 @@
 #include "llvm/CodeGen/MachineLICM.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachinePassManager.h"
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/CodeGen/MachineVerifier.h"
 #include "llvm/CodeGen/OptimizePHIs.h"
 #include "llvm/CodeGen/PHIElimination.h"
@@ -1028,7 +1029,7 @@ void CodeGenPassBuilder<Derived, TargetMachineT>::addMachineSSAOptimization(
   addPass(EarlyMachineLICMPass());
   addPass(MachineCSEPass());
 
-  addPass(MachineSinkingPass());
+  addPass(MachineSinkingPass(Opt.EnableSinkAndFold));
 
   addPass(PeepholeOptimizerPass());
   // Clean-up the dead code that may have been generated by peephole
diff --git a/llvm/include/llvm/Passes/MachinePassRegistry.def b/llvm/include/llvm/Passes/MachinePassRegistry.def
index bbd10c14e99491..041fe9c4e706d0 100644
--- a/llvm/include/llvm/Passes/MachinePassRegistry.def
+++ b/llvm/include/llvm/Passes/MachinePassRegistry.def
@@ -171,6 +171,13 @@ MACHINE_FUNCTION_PASS("verify<machine-trace-metrics>", MachineTraceMetricsVerifi
 #define MACHINE_FUNCTION_PASS_WITH_PARAMS(NAME, CLASS, CREATE_PASS, PARSER,    \
                                           PARAMS)
 #endif
+MACHINE_FUNCTION_PASS_WITH_PARAMS(
+    "machine-sink", "MachineSinkingPass",
+    [](bool EnableSinkAndFold) {
+      return MachineSinkingPass(EnableSinkAndFold);
+    },
+    parseMachineSinkingPassOptions, "enable-sink-fold")
+
 MACHINE_FUNCTION_PASS_WITH_PARAMS(
     "regallocfast", "RegAllocFast",
     [](RegAllocFastPassOptions Opts) { return RegAllocFastPass(Opts); },
@@ -178,6 +185,7 @@ MACHINE_FUNCTION_PASS_WITH_PARAMS(
       return parseRegAllocFastPassOptions(*PB, Params);
     },
     "filter=reg-filter;no-clear-vregs")
+
 #undef MACHINE_FUNCTION_PASS_WITH_PARAMS
 
 // After a pass is converted to new pass manager, its entry should be moved from
@@ -233,7 +241,6 @@ DUMMY_MACHINE_FUNCTION_PASS("machine-function-splitter", MachineFunctionSplitter
 DUMMY_MACHINE_FUNCTION_PASS("machine-latecleanup", MachineLateInstrsCleanupPass)
 DUMMY_MACHINE_FUNCTION_PASS("machine-sanmd", MachineSanitizerBinaryMetadata)
 DUMMY_MACHINE_FUNCTION_PASS("machine-scheduler", MachineSchedulerPass)
-DUMMY_MACHINE_FUNCTION_PASS("machine-sink", MachineSinkingPass)
 DUMMY_MACHINE_FUNCTION_PASS("machine-uniformity", MachineUniformityInfoWrapperPass)
 DUMMY_MACHINE_FUNCTION_PASS("machineinstr-printer", MachineFunctionPrinterPass)
 DUMMY_MACHINE_FUNCTION_PASS("mirfs-discriminators", MIRAddFSDiscriminatorsPass)
diff --git a/llvm/include/llvm/Target/CGPassBuilderOption.h b/llvm/include/llvm/Target/CGPassBuilderOption.h
index 8ab6d63a00056a..77f81902388179 100644
--- a/llvm/include/llvm/Target/CGPassBuilderOption.h
+++ b/llvm/include/llvm/Target/CGPassBuilderOption.h
@@ -32,6 +32,7 @@ struct CGPassBuilderOption {
   bool EnableImplicitNullChecks = false;
   bool EnableBlockPlacementStats = false;
   bool EnableMachineFunctionSplitter = false;
+  bool EnableSinkAndFold = false;
   bool MISchedPostRA = false;
   bool EarlyLiveIntervals = false;
   bool GCEmptyBlocks = false;
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index adddb8daaa0e91..b6881c339ff15e 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -94,7 +94,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeMachinePostDominatorTreeWrapperPassPass(Registry);
   initializeMachineRegionInfoPassPass(Registry);
   initializeMachineSchedulerPass(Registry);
-  initializeMachineSinkingPass(Registry);
+  initializeMachineSinkingLegacyPass(Registry);
   initializeMachineUniformityAnalysisPassPass(Registry);
   initializeMachineUniformityInfoPrinterPassPass(Registry);
   initializeMachineVerifierLegacyPassPass(Registry);
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index a0e09398602e9e..c06cc82cd569ac 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -15,6 +15,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/DepthFirstIterator.h"
 #include "llvm/ADT/MapVector.h"
@@ -36,6 +37,7 @@
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/MachinePassManager.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterClassInfo.h"
@@ -44,6 +46,7 @@
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/IR/Analysis.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/LLVMContext.h"
@@ -115,7 +118,7 @@ STATISTIC(NumPostRACopySink, "Number of copies sunk after RA");
 
 namespace {
 
-class MachineSinking : public MachineFunctionPass {
+class MachineSinking {
   const TargetSubtargetInfo *STI = nullptr;
   const TargetInstrInfo *TII = nullptr;
   const TargetRegisterInfo *TRI = nullptr;
@@ -127,6 +130,8 @@ class MachineSinking : public MachineFunctionPass {
   const MachineBranchProbabilityInfo *MBPI = nullptr;
   AliasAnalysis *AA = nullptr;
   RegisterClassInfo RegClassInfo;
+  Pass *LegacyPass;
+  MachineFunctionAnalysisManager *MFAM;
 
   // Remember which edges have been considered for breaking.
   SmallSet<std::pair<MachineBasicBlock *, MachineBasicBlock *>, 8>
@@ -182,29 +187,14 @@ class MachineSinking : public MachineFunctionPass {
   bool EnableSinkAndFold;
 
 public:
-  static char ID; // Pass identification
+  MachineSinking(Pass *LegacyPass, MachineFunctionAnalysisManager *MFAM,
+                 bool EnableSinkAndFold)
+      : LegacyPass(LegacyPass), MFAM(MFAM),
+        EnableSinkAndFold(EnableSinkAndFold) {}
 
-  MachineSinking() : MachineFunctionPass(ID) {
-    initializeMachineSinkingPass(*PassRegistry::getPassRegistry());
-  }
-
-  bool runOnMachineFunction(MachineFunction &MF) override;
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    MachineFunctionPass::getAnalysisUsage(AU);
-    AU.addRequired<AAResultsWrapperPass>();
-    AU.addRequired<MachineDominatorTreeWrapperPass>();
-    AU.addRequired<MachinePostDominatorTreeWrapperPass>();
-    AU.addRequired<MachineCycleInfoWrapperPass>();
-    AU.addRequired<MachineBranchProbabilityInfoWrapperPass>();
-    AU.addPreserved<MachineCycleInfoWrapperPass>();
-    AU.addPreserved<MachineLoopInfoWrapperPass>();
-    if (UseBlockFreqInfo)
-      AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
-    AU.addRequired<TargetPassConfig>();
-  }
+  bool run(MachineFunction &MF);
 
-  void releaseMemory() override {
+  void releaseMemory() {
     CEBCandidates.clear();
     CEMergeCandidates.clear();
   }
@@ -276,20 +266,45 @@ class MachineSinking : public MachineFunctionPass {
                                        const MachineBasicBlock &MBB);
 };
 
+class MachineSinkingLegacy : public MachineFunctionPass {
+public:
+  static char ID;
+
+  MachineSinkingLegacy() : MachineFunctionPass(ID) {
+    initializeMachineSinkingLegacyPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    MachineFunctionPass::getAnalysisUsage(AU);
+    AU.addRequired<AAResultsWrapperPass>();
+    AU.addRequired<MachineDominatorTreeWrapperPass>();
+    AU.addRequired<MachinePostDominatorTreeWrapperPass>();
+    AU.addRequired<MachineCycleInfoWrapperPass>();
+    AU.addRequired<MachineBranchProbabilityInfoWrapperPass>();
+    AU.addPreserved<MachineCycleInfoWrapperPass>();
+    AU.addPreserved<MachineLoopInfoWrapperPass>();
+    if (UseBlockFreqInfo)
+      AU.addRequired<MachineBlockFrequencyInfoWrapperPass>();
+    AU.addRequired<TargetPassConfig>();
+  }
+};
+
 } // end anonymous namespace
 
-char MachineSinking::ID = 0;
+char MachineSinkingLegacy::ID = 0;
 
-char &llvm::MachineSinkingID = MachineSinking::ID;
+char &llvm::MachineSinkingLegacyID = MachineSinkingLegacy::ID;
 
-INITIALIZE_PASS_BEGIN(MachineSinking, DEBUG_TYPE, "Machine code sinking", false,
-                      false)
+INITIALIZE_PASS_BEGIN(MachineSinkingLegacy, DEBUG_TYPE, "Machine code sinking",
+                      false, false)
 INITIALIZE_PASS_DEPENDENCY(MachineBranchProbabilityInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(MachineDominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(MachineCycleInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
-INITIALIZE_PASS_END(MachineSinking, DEBUG_TYPE, "Machine code sinking", false,
-                    false)
+INITIALIZE_PASS_END(MachineSinkingLegacy, DEBUG_TYPE, "Machine code sinking",
+                    false, false)
 
 /// Return true if a target defined block prologue instruction interferes
 /// with a sink candidate.
@@ -709,27 +724,57 @@ void MachineSinking::FindCycleSinkCandidates(
   }
 }
 
-bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
+PreservedAnalyses
+MachineSinkingPass::run(MachineFunction &MF,
+                        MachineFunctionAnalysisManager &MFAM) {
+  if (MF.getFunction().hasOptNone())
+    return PreservedAnalyses::all();
+
+  MachineSinking Impl(nullptr, &MFAM, EnableSinkAndFold);
+  bool Changed = Impl.run(MF);
+  if (!Changed)
+    return PreservedAnalyses::all();
+  auto PA = getMachineFunctionPassPreservedAnalyses();
+  PA.preserve<MachineCycleAnalysis>();
+  PA.preserve<MachineLoopAnalysis>();
+  return PA;
+}
+
+bool MachineSinkingLegacy::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
+  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
+  bool EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
+
+  MachineSinking Impl(this, nullptr, EnableSinkAndFold);
+  return Impl.run(MF);
+}
+
+#define GET_ANALYSIS(ANALYSIS, INFIX, GETTER)                                  \
+  ((LegacyPass)                                                                \
+       ? &LegacyPass->getAnalysis<ANALYSIS##INFIX##WrapperPass>().GETTER()     \
+       : &MFAM->getResult<ANALYSIS##Analysis>(MF))
+
+bool MachineSinking::run(MachineFunction &MF) {
   LLVM_DEBUG(dbgs() << "******** Machine Sinking ********\n");
 
   STI = &MF.getSubtarget();
   TII = STI->getInstrInfo();
   TRI = STI->getRegisterInfo();
   MRI = &MF.getRegInfo();
-  DT = &getAnalysis<MachineDominatorTreeWrapperPass>().getDomTree();
-  PDT = &getAnalysis<MachinePostDominatorTreeWrapperPass>().getPostDomTree();
-  CI = &getAnalysis<MachineCycleInfoWrapperPass>().getCycleInfo();
-  MBFI = UseBlockFreqInfo
-             ? &getAnalysis<MachineBlockFrequencyInfoWrapperPass>().getMBFI()
-             : nullptr;
-  MBPI = &getAnalysis<MachineBranchProbabilityInfoWrapperPass>().getMBPI();
-  AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+  DT = GET_ANALYSIS(MachineDominatorTree, , getDomTree);
+  PDT = GET_ANALYSIS(MachinePostDominatorTree, , getPostDomTree);
+  CI = GET_ANALYSIS(MachineCycle, Info, getCycleInfo);
+  MBFI = UseBlockFreqInfo ? GET_ANALYSIS(MachineBlockFrequency, Info, getMBFI)
+                          : nullptr;
+  MBPI = GET_ANALYSIS(MachineBranchProbability, Info, getMBPI);
+  AA = (LegacyPass)
+           ? &LegacyPass->getAnalysis<AAResultsWrapperPass>().getAAResults()
+           : &MFAM->getResult<FunctionAnalysisManagerMachineFunctionProxy>(MF)
+                  .getManager()
+                  .getResult<AAManager>(MF.getFunction());
   RegClassInfo.runOnMachineFunction(MF);
-  TargetPassConfig *PassConfig = &getAnalysis<TargetPassConfig>();
-  EnableSinkAndFold = PassConfig->getEnableSinkAndFold();
 
   bool EverMadeChange = false;
 
@@ -745,7 +790,8 @@ bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
 
     // If we have anything we marked as toSplit, split it now.
     for (const auto &Pair : ToSplit) {
-      auto NewSucc = Pair.first->SplitCriticalEdge(Pair.second, *this);
+      auto NewSucc =
+          Pair.first->SplitCriticalEdge(Pair.second, LegacyPass, MFAM, nullptr);
       if (NewSucc != nullptr) {
         LLVM_DEBUG(dbgs() << " *** Splitting critical edge: "
                           << printMBBReference(*Pair.first) << " -- "
@@ -804,6 +850,7 @@ bool MachineSinking::runOnMachineFunction(MachineFunction &MF) {
     MRI->clearKillFlags(I);
   RegsToClearKillFlags.clear();
 
+  releaseMemory();
   return EverMadeChange;
 }
 
diff --git a/llvm/lib/CodeGen/TargetPassConfig.cpp b/llvm/lib/CodeGen/TargetPassConfig.cpp
index 782ed6062191ce..be71488e39458d 100644
--- a/llvm/lib/CodeGen/TargetPassConfig.cpp
+++ b/llvm/lib/CodeGen/TargetPassConfig.cpp
@@ -317,7 +317,7 @@ static IdentifyingPassPtr overridePass(AnalysisID StandardID,
   if (StandardID == &MachineLICMID)
     return applyDisable(TargetID, DisablePostRAMachineLICM);
 
-  if (StandardID == &MachineSinkingID)
+  if (StandardID == &MachineSinkingLegacyID)
     return applyDisable(TargetID, DisableMachineSink);
 
   if (StandardID == &PostRAMachineSinkingID)
@@ -1304,7 +1304,7 @@ void TargetPassConfig::addMachineSSAOptimization() {
   addPass(&EarlyMachineLICMID);
   addPass(&MachineCSELegacyID);
 
-  addPass(&MachineSinkingID);
+  addPass(&MachineSinkingLegacyID);
 
   addPass(&PeepholeOptimizerID);
   // Clean-up the dead code that may have been generated by peephole
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index bd670c0b13df1d..b7a2c881d495c3 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -114,6 +114,7 @@
 #include "llvm/CodeGen/MachinePassManager.h"
 #include "llvm/CodeGen/MachinePostDominators.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/MachineSink.h"
 #include "llvm/CodeGen/MachineTraceMetrics.h"
 #include "llvm/CodeGen/MachineVerifier.h"
 #include "llvm/CodeGen/OptimizePHIs.h"
@@ -1277,6 +1278,11 @@ Expected<RealtimeSanitizerOptions> parseRtSanPassOptions(StringRef Params) {
   return Result;
 }
 
+Expected<bool> parseMachineSinkingPassOptions(StringRef Params) {
+  return PassBuilder::parseSinglePassOption(Params, "enable-sink-fold",
+                                            "MachineSinkingPass");
+}
+
 } // namespace
 
 /// Tests whether a pass name starts with a valid prefix for a default pipeline
diff --git a/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp b/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
index a5a147da8da1c5..20c86362ded2a6 100644
--- a/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXTargetMachine.cpp
@@ -492,7 +492,7 @@ void NVPTXPassConfig::addMachineSSAOptimization() {
   addPass(&EarlyMachineLICMID);
   addPass(&MachineCSELegacyID);
 
-  addPass(&MachineSinkingID);
+  addPass(&MachineSinkingLegacyID);
   printAndVerify("After Machine LICM, CSE and Sinking passes");
 
   addPass(&PeepholeOptimizerID);
diff --git a/llvm/test/CodeGen/AArch64/loop-sink.mir b/llvm/test/CodeGen/AArch64/loop-sink.mir
index 36d39ffbadc29a..de81d99369394b 100644
--- a/llvm/test/CodeGen/AArch64/loop-sink.mir
+++ b/llvm/test/CodeGen/AArch64/loop-sink.mir
@@ -1,5 +1,6 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple aarch64 -run-pass=machine-sink  -sink-insts-to-avoid-spills -aarch64-enable-sink-fold=true %s -o - 2>&1 | FileCheck %s
+# RUN: llc -mtriple aarch64 -passes="machine-sink<enable-sink-fold>" -sink-insts-to-avoid-spills %s -o - 2>&1 | FileCheck %s
 --- |
   target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64"
diff --git a/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir b/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
index be5737d297b290..f146c4436bbfa8 100644
--- a/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
+++ b/llvm/test/CodeGen/AArch64/sink-and-fold-preserve-debugloc.mir
@@ -1,4 +1,6 @@
 # RUN: llc %s -run-pass=machine-sink --aarch64-enable-sink-fold=true -o - | FileCheck %s
+
+# RUN: llc %s -passes="machine-sink<enable-sink-fold>" -o - | FileCheck %s
 --- |
   target triple = "aarch64-linux"
 
diff --git a/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir b/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
index 329f2967121603..7ef6d37ddc1832 100644
--- a/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
+++ b/llvm/test/CodeGen/AMDGPU/machine-sink-temporal-divergence-swdev407790.mir
@@ -1,6 +1,8 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 3
 # RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1031 -run-pass=machine-sink -o -  %s | FileCheck %s
 
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1031 -passes=machine-sink -o -  %s | FileCheck %s
+
 ---
 name: machine-sink-temporal-divergence
 tracksRegLiveness: true
diff --git a/llvm/test/CodeGen/ARM/machine-sink-multidef.mir b/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
index 737ca824265d6b..485e659a53ab6b 100644
--- a/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
+++ b/llvm/test/CodeGen/ARM/machine-sink-multidef.mir
@@ -1,6 +1,8 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc %s -o - -run-pass=machine-sink -mtriple=arm-none-eabi | FileCheck %s
 
+# RUN: llc %s -o - -passes=machine-sink -mtriple=arm-none-eabi | FileCheck %s
+
 --- |
   target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
   target triple = "arm-none-unknown-eabi"
diff --git a/llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir b/llvm/test/CodeGen/Hexagon/machine-sink-float-usr.mir
i...
[truncated]

optimisan · 2024-11-08T07:29:52Z

I've updated a few tests for all targets since there are a lot of machine-sink tests.

llvm/include/llvm/CodeGen/MachineSink.h

cjappl · 2024-11-08T15:55:34Z

(sorry I don't know anything about this part of the codebase, so I am removing myself from the reviewers)

arsenm · 2024-11-08T16:58:14Z

llvm/lib/CodeGen/MachineSink.cpp

+  Pass *LegacyPass;
+  MachineFunctionAnalysisManager *MFAM;


Keep pass manager details out of the implementation

The other alternative is passing all analyses as parameters and five seemed too many

That is what you should do, and what other passes do

@optimisan this is pending.

SplitCriticalEdge requires either Pass or MFAM, so I put them here. I'll remove that dependency first then.

llvm/lib/CodeGen/MachineSink.cpp

optimisan · 2024-11-19T12:30:40Z

Sorry, I have no idea where code owners request came from!

llvm/include/llvm/Passes/MachinePassRegistry.def

cdevadas · 2024-12-20T05:17:00Z

llvm/lib/CodeGen/MachineSink.cpp

+  Pass *LegacyPass;
+  MachineFunctionAnalysisManager *MFAM;


@optimisan this is pending.

llvm/lib/CodeGen/MachineSink.cpp

cdevadas · 2025-02-21T15:36:28Z

llvm/lib/CodeGen/MachineSink.cpp

+                 MachineCycleInfo *CI, ProfileSummaryInfo *PSI,
+                 MachineBlockFrequencyInfo *MBFI,
+                 const MachineBranchProbabilityInfo *MBPI, AliasAnalysis *AA)
+      : DT(DT), PDT(PDT), CI(CI), PSI(PSI), MBFI(MBFI), MBPI(MBPI), AA(AA),


Should have used RequiredAnalyses instead of this long list of arguments.

Figured it's not essential, later we can just collapse impl into the new pass class.

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

optimisan mentioned this pull request Nov 8, 2024

[NFC][CodeGen] Clang format MachineSink.cpp #114027

Merged

This was referenced Nov 8, 2024

[CodeGen][NewPM] Port MachineCycleInfo to NPM #114745

Merged

[CodeGen] Move EnableSinkAndFold to TargetOptions #114746

Closed

optimisan marked this pull request as ready for review November 8, 2024 07:28

optimisan requested a review from paperchalice November 8, 2024 07:28

llvmbot added backend:ARM backend:AArch64 backend:AMDGPU backend:Hexagon backend:PowerPC backend:SystemZ backend:X86 debuginfo backend:NVPTX labels Nov 8, 2024

optimisan requested review from cjappl and cdevadas November 8, 2024 07:29

paperchalice reviewed Nov 8, 2024

View reviewed changes

llvm/include/llvm/CodeGen/MachineSink.h Show resolved Hide resolved

cjappl removed their request for review November 8, 2024 15:55

shiltian requested review from nikic and arsenm November 8, 2024 15:57

arsenm reviewed Nov 8, 2024

View reviewed changes

paperchalice reviewed Nov 9, 2024

View reviewed changes

llvm/lib/CodeGen/MachineSink.cpp Outdated Show resolved Hide resolved

optimisan force-pushed the users/Akshat-Oke/10-30-_codegen_newpm_port_machinecycleinfo_to_npm branch from 7972323 to 68bcb36 Compare November 14, 2024 06:00

optimisan mentioned this pull request Nov 14, 2024

[NewPM] Introduce MFAnalysisGetter for a common analysis getter #116166

Closed

optimisan removed request for banach-space, rengolin, JDevlieghere, lanza, bcardosolopes, nikic and a team November 19, 2024 12:28

cdevadas reviewed Dec 20, 2024

View reviewed changes

optimisan force-pushed the users/Akshat-Oke/10-29-_codegen_newpm_port_machinesink_to_npm branch from 1b41043 to 6f444ce Compare February 21, 2025 06:32

optimisan force-pushed the users/Akshat-Oke/10-30-_codegen_newpm_port_machinecycleinfo_to_npm branch from 6236214 to c366c19 Compare February 21, 2025 06:32

optimisan mentioned this pull request Feb 21, 2025

[MachineBasicBlock][NFC] Decouple SplitCriticalEdges from pass manager #128151

Merged

optimisan force-pushed the users/Akshat-Oke/10-29-_codegen_newpm_port_machinesink_to_npm branch from 6f444ce to fd3300b Compare February 21, 2025 09:31

optimisan changed the base branch from users/Akshat-Oke/10-30-_codegen_newpm_port_machinecycleinfo_to_npm to users/Akshat-Oke/decouple-mbb-splitcriticaledge February 21, 2025 09:31

arsenm approved these changes Feb 21, 2025

View reviewed changes

optimisan force-pushed the users/Akshat-Oke/10-29-_codegen_newpm_port_machinesink_to_npm branch from b310623 to a728204 Compare February 21, 2025 09:47

cdevadas reviewed Feb 21, 2025

View reviewed changes

optimisan force-pushed the users/Akshat-Oke/decouple-mbb-splitcriticaledge branch from 24b2204 to dbdc66e Compare February 24, 2025 09:32

Base automatically changed from users/Akshat-Oke/decouple-mbb-splitcriticaledge to main February 25, 2025 04:40

optimisan added 5 commits March 3, 2025 06:56

[CodeGen][NewPM] Port MachineSink to NPM

1d24833

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

remove PM from impl class

c9cbc85

remove macro

06fc1e7

remove superfluous includes

b09cfdf

clang format

91b8868

optimisan force-pushed the users/Akshat-Oke/10-29-_codegen_newpm_port_machinesink_to_npm branch from a728204 to 91b8868 Compare March 3, 2025 07:51

arsenm approved these changes Mar 3, 2025

View reviewed changes

fix analyses required for split critical edge

53a87f3

optimisan merged commit 77f44a9 into main Mar 3, 2025
8 of 11 checks passed

optimisan deleted the users/Akshat-Oke/10-29-_codegen_newpm_port_machinesink_to_npm branch March 3, 2025 10:19

jph-13 pushed a commit to jph-13/llvm-project that referenced this pull request Mar 21, 2025

[CodeGen][NewPM] Port MachineSink to NPM (llvm#115434)

ced0fbb

Targets can set the EnableSinkAndFold option in CGPassBuilderOptions for the NPM pipeline in buildCodeGenPipeline(... &Opts, ...)

[CodeGen][NewPM] Port MachineSink to NPM #115434

[CodeGen][NewPM] Port MachineSink to NPM #115434

Uh oh!

Conversation

optimisan commented Nov 8, 2024

Uh oh!

optimisan commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Nov 8, 2024

Uh oh!

llvmbot commented Nov 8, 2024

Uh oh!

optimisan commented Nov 8, 2024

Uh oh!

Uh oh!

cjappl commented Nov 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

optimisan commented Nov 19, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

optimisan commented Nov 8, 2024 •

edited

Loading

llvmbot commented Nov 8, 2024 •

edited

Loading