[LoopVectorize] Further improve cost model for early exit loops #126235

david-arm · 2025-02-07T11:52:45Z

Following on from #125058, this patch takes into account the
work done in the vector early exit block when assessing the
profitability of vectorising the loop. I have renamed
areRuntimeChecksProfitable to isOutsideLoopWorkProfitable and
we now pass in the early exit costs. As part of this, I have
added the ExtractFirstActive opcode to VPInstruction::computeCost.

It's worth pointing out that when we assess profitability of the
loop we calculate a minimum trip count and compare that against
the maximum trip count. However, since the loop has an early
exit the runtime trip count can still end up being less than the
minimum. Alternatively, we may never take the early exit at all
at runtime and so we have the opposite problem of over-estimating
the cost of the loop. The loop vectoriser cannot simultaneously
take two contradictory positions and so I feel the only sensible
thing to do is be conservative and assume the loop will be more
expensive than loops without early exits.

We may find in future that we need to adjust the cost according to
the probability of taking the early exit. This will become even
more important once we support multiple early exits. However, we
have to start somewhere and we can always revisit this later.

llvmbot · 2025-02-07T11:53:19Z

@llvm/pr-subscribers-vectorizers

@llvm/pr-subscribers-llvm-transforms

Author: David Sherwood (david-arm)

Changes

Following on from #125058, this patch takes into account the
work done in the vector early exit block when assessing the
profitability of vectorising the loop. I have renamed
areRuntimeChecksProfitable to isOutsideLoopWorkProfitable and
we now pass in the early exit costs. As part of this, I have
added the ExtractFirstActive opcode to VPInstruction::computeCost.

It's worth pointing out that when we assess profitability of the
loop we calculate a minimum trip count and compare that against
the maximum trip count. However, since the loop has an early
exit the runtime trip count can still end up being less than the
minimum. Alternatively, we may never take the early exit at all
at runtime and so we have the opposite problem of over-estimating
the cost of the loop. The loop vectoriser cannot simultaneously
take two contradictory positions and so I feel the only sensible
thing to do is be conservative and assume the loop will be more
expensive than loops without early exits.

We may find in future that we need to adjust the cost according to
the probability of taking the early exit. This will become even
more important once we support multiple early exits. However, we
have to start somewhere and we can always revisit this later.

Full diff: https://github.com/llvm/llvm-project/pull/126235.diff

4 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+44-12)
(modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+13)
(added) llvm/test/Transforms/LoopVectorize/AArch64/early_exit_costs.ll (+47)
(modified) llvm/test/Transforms/LoopVectorize/AArch64/simple_early_exit.ll (+1)

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 6ad44259ccdf6e..7538dbe07a6466 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -10077,19 +10077,46 @@ static void checkMixedPrecision(Loop *L, OptimizationRemarkEmitter *ORE) {
   }
 }
 
-static bool areRuntimeChecksProfitable(GeneratedRTChecks &Checks,
-                                       VectorizationFactor &VF, Loop *L,
-                                       PredicatedScalarEvolution &PSE,
-                                       ScalarEpilogueLowering SEL,
-                                       std::optional<unsigned> VScale) {
+static InstructionCost calculateEarlyExitCost(LoopVectorizationCostModel &CM,
+                                              VPlan &Plan, ElementCount VF) {
+  InstructionCost Cost = 0;
+  VPCostContext CostCtx(CM.TTI, *CM.TLI, CM.Legal->getWidestInductionType(), CM,
+                        CM.CostKind);
+  LLVM_DEBUG(
+      dbgs() << "Calculating cost of work in vector early exit block:\n");
+  for (auto *ExitVPBB : Plan.getExitBlocks()) {
+    for (auto *PredVPBB : ExitVPBB->getPredecessors())
+      if (PredVPBB != Plan.getMiddleBlock())
+        for (auto &R : *(cast<VPBasicBlock>(PredVPBB)))
+          Cost += R.cost(VF, CostCtx);
+  }
+  return Cost;
+}
+
+static bool isOutsideLoopWorkProfitable(GeneratedRTChecks &Checks,
+                                        VectorizationFactor &VF, Loop *L,
+                                        const TargetTransformInfo &TTI,
+                                        PredicatedScalarEvolution &PSE,
+                                        ScalarEpilogueLowering SEL,
+                                        std::optional<unsigned> VScale,
+                                        InstructionCost EarlyExitCost) {
   InstructionCost CheckCost = Checks.getCost();
-  if (!CheckCost.isValid())
+  if (!CheckCost.isValid() && !EarlyExitCost.isValid())
     return false;
 
+  InstructionCost TotalCost = 0;
+  if (CheckCost.isValid())
+    TotalCost += CheckCost;
+
+  // Add on the cost of work required in the vector early exit block, if one
+  // exists.
+  if (EarlyExitCost.isValid())
+    TotalCost += EarlyExitCost;
+
   // When interleaving only scalar and vector cost will be equal, which in turn
   // would lead to a divide by 0. Fall back to hard threshold.
   if (VF.Width.isScalar()) {
-    if (CheckCost > VectorizeMemoryCheckThreshold) {
+    if (TotalCost > VectorizeMemoryCheckThreshold) {
       LLVM_DEBUG(
           dbgs()
           << "LV: Interleaving only is not profitable due to runtime checks\n");
@@ -10132,7 +10159,7 @@ static bool areRuntimeChecksProfitable(GeneratedRTChecks &Checks,
   // the computations are performed on doubles, not integers and the result
   // is rounded up, hence we get an upper estimate of the TC.
   unsigned IntVF = getEstimatedRuntimeVF(VF.Width, VScale);
-  uint64_t RtC = *CheckCost.getValue();
+  uint64_t RtC = *TotalCost.getValue();
   uint64_t Div = ScalarC * IntVF - *VF.Cost.getValue();
   uint64_t MinTC1 = Div == 0 ? 0 : divideCeil(RtC * IntVF, Div);
 
@@ -10468,8 +10495,8 @@ bool LoopVectorizePass::processLoop(Loop *L) {
         // iteration count is low. However, setting the epilogue policy to
         // `CM_ScalarEpilogueNotAllowedLowTripLoop` prevents vectorizing loops
         // with runtime checks. It's more effective to let
-        // `areRuntimeChecksProfitable` determine if vectorization is beneficial
-        // for the loop.
+        // `isOutsideLoopWorkProfitable` determine if vectorization is
+        // beneficial for the loop.
         if (SEL != CM_ScalarEpilogueNotNeededUsePredicate)
           SEL = CM_ScalarEpilogueNotAllowedLowTripLoop;
       } else {
@@ -10564,12 +10591,17 @@ bool LoopVectorizePass::processLoop(Loop *L) {
     if (VF.Width.isVector() || SelectedIC > 1)
       Checks.create(L, *LVL.getLAI(), PSE.getPredicate(), VF.Width, SelectedIC);
 
+    InstructionCost EarlyExitCost = InstructionCost::getInvalid();
+    if (VF.Width.isVector() && LVL.hasUncountableEarlyExit())
+      EarlyExitCost =
+          calculateEarlyExitCost(CM, LVP.getPlanFor(VF.Width), VF.Width);
+
     // Check if it is profitable to vectorize with runtime checks.
     bool ForceVectorization =
         Hints.getForce() == LoopVectorizeHints::FK_Enabled;
     if (!ForceVectorization &&
-        !areRuntimeChecksProfitable(Checks, VF, L, PSE, SEL,
-                                    CM.getVScaleForTuning())) {
+        !isOutsideLoopWorkProfitable(Checks, VF, L, *TTI, PSE, SEL,
+                                     CM.getVScaleForTuning(), EarlyExitCost)) {
       ORE->emit([&]() {
         return OptimizationRemarkAnalysisAliasing(
                    DEBUG_TYPE, "CantReorderMemOps", L->getStartLoc(),
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index a1b78fa244e17a..6ca82ef99a23bc 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -726,6 +726,19 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
     return Ctx.TTI.getArithmeticReductionCost(
         Instruction::Or, cast<VectorType>(VecTy), std::nullopt, Ctx.CostKind);
   }
+  case VPInstruction::ExtractFirstActive: {
+    // Calculate the cost of determining the lane index.
+    auto *PredTy = toVectorTy(Ctx.Types.inferScalarType(getOperand(1)), VF);
+    IntrinsicCostAttributes Attrs(
+        Intrinsic::experimental_cttz_elts, Type::getInt64Ty(Ctx.LLVMCtx),
+        {PoisonValue::get(PredTy), ConstantInt::getTrue(Ctx.LLVMCtx)});
+    InstructionCost Cost = Ctx.TTI.getIntrinsicInstrCost(Attrs, Ctx.CostKind);
+    // Add on the cost of extracting the element.
+    auto *VecTy = toVectorTy(Ctx.Types.inferScalarType(getOperand(0)), VF);
+    Cost += Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,
+                                       Ctx.CostKind);
+    return Cost;
+  }
   default:
     // TODO: Fill out other opcodes!
     return 0;
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/early_exit_costs.ll b/llvm/test/Transforms/LoopVectorize/AArch64/early_exit_costs.ll
new file mode 100644
index 00000000000000..d38d1f04b1e8de
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/early_exit_costs.ll
@@ -0,0 +1,47 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 4
+; REQUIRES: asserts
+; RUN: opt -S < %s -p loop-vectorize -enable-early-exit-vectorization -disable-output \
+; RUN:   -debug-only=loop-vectorize 2>&1 | FileCheck %s --check-prefixes=CHECK
+
+target triple = "aarch64-unknown-linux-gnu"
+
+declare void @init_mem(ptr, i64);
+
+define i64 @same_exit_block_pre_inc_use1() #1 {
+; CHECK-LABEL: LV: Checking a loop in 'same_exit_block_pre_inc_use1'
+; CHECK: LV: Selecting VF: vscale x 16
+; CHECK: Calculating cost of work in vector early exit block:
+; CHECK-NEXT: Cost of 6 for VF vscale x 16: EMIT vp<{{.*}}> = extract-first-active
+; CHECK-NEXT: Cost of 6 for VF vscale x 16: EMIT vp<{{.*}}> = extract-first-active
+; CHECK: LV: Minimum required TC for runtime checks to be profitable:32
+entry:
+  %p1 = alloca [1024 x i8]
+  %p2 = alloca [1024 x i8]
+  call void @init_mem(ptr %p1, i64 1024)
+  call void @init_mem(ptr %p2, i64 1024)
+  br label %loop
+
+loop:
+  %index = phi i64 [ %index.next, %loop.inc ], [ 3, %entry ]
+  %index2 = phi i64 [ %index2.next, %loop.inc ], [ 15, %entry ]
+  %arrayidx = getelementptr inbounds i8, ptr %p1, i64 %index
+  %ld1 = load i8, ptr %arrayidx, align 1
+  %arrayidx1 = getelementptr inbounds i8, ptr %p2, i64 %index
+  %ld2 = load i8, ptr %arrayidx1, align 1
+  %cmp3 = icmp eq i8 %ld1, %ld2
+  br i1 %cmp3, label %loop.inc, label %loop.end
+
+loop.inc:
+  %index.next = add i64 %index, 1
+  %index2.next = add i64 %index2, 2
+  %exitcond = icmp ne i64 %index.next, 67
+  br i1 %exitcond, label %loop, label %loop.end
+
+loop.end:
+  %val1 = phi i64 [ %index, %loop ], [ 67, %loop.inc ]
+  %val2 = phi i64 [ %index2, %loop ], [ 98, %loop.inc ]
+  %retval = add i64 %val1, %val2
+  ret i64 %retval
+}
+
+attributes #1 = { "target-features"="+sve" vscale_range(1,16) }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/simple_early_exit.ll b/llvm/test/Transforms/LoopVectorize/AArch64/simple_early_exit.ll
index b439b64e829e5f..1b4172c58a034f 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/simple_early_exit.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/simple_early_exit.ll
@@ -274,6 +274,7 @@ define i64 @loop_contains_safe_div() #1 {
 ; CHECK-NEXT:    call void @init_mem(ptr [[P2]], i64 1024)
 ; CHECK-NEXT:    [[TMP11:%.*]] = call i64 @llvm.vscale.i64()
 ; CHECK-NEXT:    [[TMP12:%.*]] = mul i64 [[TMP11]], 4
+; CHECK-NEXT:    [[TMP18:%.*]] = call i64 @llvm.umax.i64(i64 12, i64 [[TMP12]])
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    [[TMP10:%.*]] = call i64 @llvm.vscale.i64()

david-arm · 2025-02-21T09:39:49Z

Gentle ping. :)

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

fhahn · 2025-02-26T10:28:34Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

-                                       PredicatedScalarEvolution &PSE,
-                                       ScalarEpilogueLowering SEL,
-                                       std::optional<unsigned> VScale) {
+static InstructionCost calculateEarlyExitCost(LoopVectorizationCostModel &CM,


Could you document this?

fhahn · 2025-02-26T10:29:21Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  return Cost;
+}
+
+static bool isOutsideLoopWorkProfitable(GeneratedRTChecks &Checks,


Could you document this, now that this does more than checking runtime checks?

fhahn · 2025-02-26T10:33:33Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+    auto *PredTy = toVectorTy(Ctx.Types.inferScalarType(getOperand(1)), VF);
+    IntrinsicCostAttributes Attrs(
+        Intrinsic::experimental_cttz_elts, Type::getInt64Ty(Ctx.LLVMCtx),
+        {PoisonValue::get(PredTy), ConstantInt::getTrue(Ctx.LLVMCtx)});


Are the arguments here only used to pass the type? Is there a variant of IntrinsicCostAttributes that just takes the argument types?

Yep, you're right. Good suggestion!

fhahn · 2025-02-26T10:33:50Z

llvm/test/Transforms/LoopVectorize/AArch64/early_exit_costs.ll

+
+declare void @init_mem(ptr, i64);
+
+define i64 @same_exit_block_pre_inc_use1() #1 {


Can you also add a test for non-SVE?

fhahn · 2025-02-26T10:37:57Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -10132,7 +10159,7 @@ static bool areRuntimeChecksProfitable(GeneratedRTChecks &Checks,
  // the computations are performed on doubles, not integers and the result
  // is rounded up, hence we get an upper estimate of the TC.
  unsigned IntVF = getEstimatedRuntimeVF(VF.Width, VScale);
-  uint64_t RtC = *CheckCost.getValue();
+  uint64_t RtC = *TotalCost.getValue();


The reference to RtC in the comment above needs updating as well

fhahn · 2025-02-26T10:45:06Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -10564,12 +10591,17 @@ bool LoopVectorizePass::processLoop(Loop *L) {
    if (VF.Width.isVector() || SelectedIC > 1)
      Checks.create(L, *LVL.getLAI(), PSE.getPredicate(), VF.Width, SelectedIC);

+    InstructionCost EarlyExitCost = InstructionCost::getInvalid();
+    if (VF.Width.isVector() && LVL.hasUncountableEarlyExit())


Do we need to check if VF is a vector? At the moment, we won't generate interleaved VPlans with early exits, but even when we do, it should probably calculate the correct cost after the VPlan has been unrolled explicitly?

fhahn · 2025-02-26T10:47:01Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+    InstructionCost EarlyExitCost = InstructionCost::getInvalid();
+    if (VF.Width.isVector() && LVL.hasUncountableEarlyExit())
+      EarlyExitCost =
+          calculateEarlyExitCost(CM, LVP.getPlanFor(VF.Width), VF.Width);


Would it be possible to instead pass the plan to isOutsideLoopWorkProfitable and compute the cost if needed there?

david-arm · 2025-03-04T09:32:13Z

Gentle ping. :)

fhahn · 2025-03-04T09:41:20Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+        for (auto &R : *(cast<VPBasicBlock>(PredVPBB)))
+          Cost += R.cost(VF, CostCtx);


Should be able to use VPBlockBase::cost?

Suggested change

for (auto &R : *(cast<VPBasicBlock>(PredVPBB)))

Cost += R.cost(VF, CostCtx);

Cost += PredVPBB->cost(VF, CostCtx)`

fhahn · 2025-03-04T09:47:37Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+    Cost += Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,
+                                       Ctx.CostKind);
+    return Cost;


Simpler to directly return?

Suggested change

Cost += Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,

Ctx.CostKind);

return Cost;

return Cost + Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,

Ctx.CostKind);

fhahn · 2025-03-04T13:05:58Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  // Add on the cost of work required in the vector early exit block, if one
+  // exists.
+  if (CM.Legal->hasUncountableEarlyExit())
+    TotalCost += calculateEarlyExitCost(CM, Plan, VF.Width);


Can this be done unconditionally so we don't have to pull in a dependency on LoopVectorizationCostModel? Would be good if we could avoid adding new uses, to not make it harder to remove.

Done. Note I had to update some tests because of the extra debug output.

david-arm · 2025-03-05T15:49:15Z

Rebased due to test conflicts

fhahn · 2025-03-05T20:01:39Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -10258,7 +10298,7 @@ static bool areRuntimeChecksProfitable(GeneratedRTChecks &Checks,

  // Skip vectorization if the expected trip count is less than the minimum
  // required trip count.
-  if (auto ExpectedTC = getSmallBestKnownTC(PSE, L)) {
+  if (auto ExpectedTC = getSmallBestKnownTC(PSE, CM.TheLoop)) {


Again, it is related because we don't have a L function argument.

fhahn · 2025-03-05T20:01:55Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -10230,8 +10270,8 @@ static bool areRuntimeChecksProfitable(GeneratedRTChecks &Checks,
  // For now we assume the epilogue cost EpiC = 0 for simplicity. Note that
  // the computations are performed on doubles, not integers and the result
  // is rounded up, hence we get an upper estimate of the TC.
-  unsigned IntVF = getEstimatedRuntimeVF(VF.Width, VScale);
-  uint64_t RtC = *CheckCost.getValue();
+  unsigned IntVF = getEstimatedRuntimeVF(VF.Width, CM.getVScaleForTuning());


Not sure what you mean by 'unrelated'? Are you referring to the s/VScale/CM.getVScaleForTuning()/? If so, this is related to the patch since I'm passing in LoopVectorizationCostModel, and have removed the VScale function argument.

fhahn · 2025-03-05T20:02:22Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -10659,8 +10699,8 @@ bool LoopVectorizePass::processLoop(Loop *L) {
    bool ForceVectorization =
        Hints.getForce() == LoopVectorizeHints::FK_Enabled;
    if (!ForceVectorization &&
-        !areRuntimeChecksProfitable(Checks, VF, L, PSE, SEL,
-                                    CM.getVScaleForTuning())) {
+        !isOutsideLoopWorkProfitable(Checks, VF, CM, PSE,


passing CM is now unnecessary/unrelated?

Oh I read this after reading your 'unrelated' comments above, which I found confusing at first. The problem is that so many of the objects required to calculate the early exit cost live in the cost model class so it seemed simpler to pass that in here. Perhaps I'm missing something, but it wasn't clear from your comments above what you wanted me to do here? I thought originally you were suggesting I move the creation of the VPCostContext object into isOutsideLoopWorkProfitable, but are you suggesting I create the VPCostContext class here instead and pass it to isOutsideLoopWorkProfitable?

Adding here was my thinking, to avoid spreading further references of the legacy CM.

fhahn · 2025-03-05T20:03:11Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  LLVM_DEBUG(
+      dbgs() << "Calculating cost of work in vector early exit block:\n");


Can we print this only if there's an early exit block?

We can, but then we want to print it before calling the cost function otherwise it would look weird with debug output like:

Cost of ...
Calculating cost of work in vector early exit block

So the only way to do this is to duplicate the loop below and check for an early exit. I can collect all the blocks in a vector and check for non-zero size I suppose. This debug printing issue is actually the reason I used to guard calling this function by querying Legal->hasUncountableEarlyExit(), but now that's been removed we unconditionally call this function.

Could the print be moved to

if (PredVPBB != Plan.getMiddleBlock()) Cost += PredVPBB->cost(VF, CostCtx);

?

Currently the wording implies there's a single block, but could also include the name of the earlier exit block, as it should work for any number of exit blocks?

Yep, good suggestion. Thanks!

fhahn · 2025-03-05T20:04:45Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+static InstructionCost calculateEarlyExitCost(LoopVectorizationCostModel &CM,
+                                              VPlan &Plan, ElementCount VF) {
+  InstructionCost Cost = 0;
+  VPCostContext CostCtx(CM.TTI, *CM.TLI, CM.Legal->getWidestInductionType(), CM,


Could we pass the cost context instead of explicitly threading through the legacy cost model, to limit new uses?

david-arm · 2025-03-10T08:53:10Z

Hi @fhahn, could you clarify what you mean by passing in CostCtx? I have addressed most of your review comments downstream, but it's not clear whether you want me to just pass it into calculateEarlyExitCost and create it in isOutsideLoopWorkProfitable, or pass it to isOutsideLoopWorkProfitable. Once I know your preference I can upload a new version - thanks!

fhahn

Hi @fhahn, could you clarify what you mean by passing in CostCtx? I have addressed most of your review comments downstream, but it's not clear whether you want me to just pass it into calculateEarlyExitCost and create it in isOutsideLoopWorkProfitable, or pass it to isOutsideLoopWorkProfitable. Once I know your preference I can upload a new version - thanks!

My thinking was pass to isOutsideLoopWorkProfitable instead of the legacy cost model, to avoid spreading new references of it

fhahn · 2025-03-10T10:24:56Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  LLVM_DEBUG(
+      dbgs() << "Calculating cost of work in vector early exit block:\n");


Could the print be moved to

if (PredVPBB != Plan.getMiddleBlock()) Cost += PredVPBB->cost(VF, CostCtx);

?

Currently the wording implies there's a single block, but could also include the name of the earlier exit block, as it should work for any number of exit blocks?

fhahn · 2025-03-10T10:25:53Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

@@ -10659,8 +10699,8 @@ bool LoopVectorizePass::processLoop(Loop *L) {
    bool ForceVectorization =
        Hints.getForce() == LoopVectorizeHints::FK_Enabled;
    if (!ForceVectorization &&
-        !areRuntimeChecksProfitable(Checks, VF, L, PSE, SEL,
-                                    CM.getVScaleForTuning())) {
+        !isOutsideLoopWorkProfitable(Checks, VF, CM, PSE,


Adding here was my thinking, to avoid spreading further references of the legacy CM.

david-arm · 2025-03-10T13:02:59Z

My thinking was pass to isOutsideLoopWorkProfitable instead of the legacy cost model, to avoid spreading new references of it

Perfect, thanks for clarifying!

fhahn

LGTM, thanks.

Please strip the changes in llvm/test/Transforms/LoopVectorize/AArch64/low_trip_memcheck_cost.ll if they aren't needed, which I would expect with the latest changes

fhahn · 2025-03-10T21:18:43Z

llvm/test/Transforms/LoopVectorize/AArch64/low_trip_memcheck_cost.ll

@@ -8,7 +8,7 @@ define void @no_outer_loop(ptr nocapture noundef %a, ptr nocapture noundef reado
 ; CHECK:      Calculating cost of runtime checks:
 ; CHECK-NOT:  We expect runtime memory checks to be hoisted out of the outer loop.
 ; CHECK:      Total cost of runtime checks: 4
-; CHECK-NEXT: LV: Minimum required TC for runtime checks to be profitable:16
+; CHECK:      LV: Minimum required TC for runtime checks to be profitable:16


Those changes shouldn't be needed with the latest changes I think?

Following on from llvm#125058, this patch takes into account the work done in the vector early exit block when assessing the profitability of vectorising the loop. I have renamed areRuntimeChecksProfitable to isOutsideLoopWorkProfitable and we now pass in the early exit costs. As part of this, I have added the ExtractFirstActive opcode to VPInstruction::computeCost. It's worth pointing out that when we assess profitability of the loop we calculate a minimum trip count and compare that against the *maximum* trip count. However, since the loop has an early exit the runtime trip count can still end up being less than the minimum. Alternatively, we may never take the early exit at all at runtime and so we have the opposite problem of over-estimating the cost of the loop. The loop vectoriser cannot simultaneously take two contradictory positions and so I feel the only sensible thing to do is be conservative and assume the loop will be more expensive than loops without early exits. We may find in future that we need to adjust the cost according to the probability of taking the early exit. This will become even more important once we support multiple early exits. However, we have to start somewhere and we can always revisit this later.

david-arm · 2025-03-11T11:48:38Z

Linux builds seem to be stuck, but the Windows build has passed (and I ran make check-all downstream) so I'll merge it.

david-arm requested review from fhahn, hassnaaHamdi and huntergr-arm February 7, 2025 11:52

llvmbot added vectorizers llvm:transforms labels Feb 7, 2025

david-arm requested a review from MacDue February 26, 2025 09:45

MacDue reviewed Feb 26, 2025

View reviewed changes

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp Outdated Show resolved Hide resolved

fhahn reviewed Feb 26, 2025

View reviewed changes

david-arm force-pushed the ee_costs2 branch from 629331a to 4ca3e8e Compare February 27, 2025 09:32

fhahn reviewed Mar 4, 2025

View reviewed changes

david-arm force-pushed the ee_costs2 branch from 67fe6fa to 8e6e54a Compare March 5, 2025 15:48

fhahn reviewed Mar 5, 2025

View reviewed changes

fhahn reviewed Mar 10, 2025

View reviewed changes

fhahn approved these changes Mar 10, 2025

View reviewed changes

david-arm added 6 commits March 11, 2025 09:55

Address review comments

33272b0

Address review comments

0e082a2

Fix test after rebase

b22497f

Address review comments

4afbfd0

Address review comment

08c1f5b

david-arm force-pushed the ee_costs2 branch from 159ae39 to 08c1f5b Compare March 11, 2025 09:57

david-arm merged commit 26ecf97 into llvm:main Mar 11, 2025
9 of 10 checks passed

david-arm deleted the ee_costs2 branch April 7, 2025 16:07


		declare void @init_mem(ptr, i64);

		define i64 @same_exit_block_pre_inc_use1() #1 {

		for (auto &R : *(cast<VPBasicBlock>(PredVPBB)))
		Cost += R.cost(VF, CostCtx);

	for (auto &R : *(cast<VPBasicBlock>(PredVPBB)))
	Cost += R.cost(VF, CostCtx);
	Cost += PredVPBB->cost(VF, CostCtx)`

		LLVM_DEBUG(
		dbgs() << "Calculating cost of work in vector early exit block:\n");

[LoopVectorize] Further improve cost model for early exit loops #126235

[LoopVectorize] Further improve cost model for early exit loops #126235

Uh oh!

Conversation

david-arm commented Feb 7, 2025

Uh oh!

llvmbot commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

david-arm commented Feb 21, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-arm commented Mar 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-arm commented Mar 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

llvmbot commented Feb 7, 2025 •

edited

Loading