Skip to content

Commit 8bb3607

Browse files
committed
[SLP] Pass VecPred argument to getCmpSelInstrCost.
Check if all compares in VL have the same predicate and pass it to getCmpSelInstrCost, to improve cost-modeling on targets that only support compare/select combinations for certain uniform predicates. This leads to additional vectorization in some cases ``` Same hash: 217 (filtered out) Remaining: 19 Metric: SLP.NumVectorInstructions Program base slp2 diff test-suite...marks/SciMark2-C/scimark2.test 11.00 26.00 136.4% test-suite...T2006/445.gobmk/445.gobmk.test 79.00 135.00 70.9% test-suite...ediabench/gsm/toast/toast.test 54.00 71.00 31.5% test-suite...telecomm-gsm/telecomm-gsm.test 54.00 71.00 31.5% test-suite...CI_Purple/SMG2000/smg2000.test 426.00 542.00 27.2% test-suite...ch/g721/g721encode/encode.test 30.00 24.00 -20.0% test-suite...000/186.crafty/186.crafty.test 116.00 138.00 19.0% test-suite...ications/JM/ldecod/ldecod.test 697.00 765.00 9.8% test-suite...6/464.h264ref/464.h264ref.test 822.00 886.00 7.8% test-suite...chmarks/MallocBench/gs/gs.test 154.00 162.00 5.2% test-suite...nsumer-lame/consumer-lame.test 621.00 651.00 4.8% test-suite...lications/ClamAV/clamscan.test 223.00 231.00 3.6% test-suite...marks/7zip/7zip-benchmark.test 680.00 695.00 2.2% test-suite...CFP2000/177.mesa/177.mesa.test 2121.00 2129.00 0.4% test-suite...:: External/Povray/povray.test 2406.00 2412.00 0.2% test-suite...TimberWolfMC/timberwolfmc.test 634.00 634.00 0.0% test-suite...CFP2006/433.milc/433.milc.test 1036.00 1036.00 0.0% test-suite.../Benchmarks/nbench/nbench.test 321.00 321.00 0.0% test-suite...ctions-flt/Reductions-flt.test NaN 5.00 nan% ``` Reviewed By: RKSimon Differential Revision: https://reviews.llvm.org/D90124
1 parent c259be9 commit 8bb3607

File tree

4 files changed

+296
-639
lines changed

4 files changed

+296
-639
lines changed

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3455,9 +3455,25 @@ int BoUpSLP::getEntryCost(TreeEntry *E) {
34553455
}
34563456
auto *MaskTy = FixedVectorType::get(Builder.getInt1Ty(), VL.size());
34573457
int ScalarCost = VecTy->getNumElements() * ScalarEltCost;
3458-
int VecCost =
3459-
TTI->getCmpSelInstrCost(E->getOpcode(), VecTy, MaskTy,
3460-
CmpInst::BAD_ICMP_PREDICATE, CostKind, VL0);
3458+
3459+
// Check if all entries in VL are either compares or selects with compares
3460+
// as condition that have the same predicates.
3461+
CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE;
3462+
bool First = true;
3463+
for (auto *V : VL) {
3464+
CmpInst::Predicate CurrentPred;
3465+
auto MatchCmp = m_Cmp(CurrentPred, m_Value(), m_Value());
3466+
if ((!match(V, m_Select(MatchCmp, m_Value(), m_Value())) &&
3467+
!match(V, MatchCmp)) ||
3468+
(!First && VecPred != CurrentPred)) {
3469+
VecPred = CmpInst::BAD_ICMP_PREDICATE;
3470+
break;
3471+
}
3472+
First = false;
3473+
VecPred = CurrentPred;
3474+
}
3475+
int VecCost = TTI->getCmpSelInstrCost(E->getOpcode(), VecTy, MaskTy,
3476+
VecPred, CostKind, VL0);
34613477
return ReuseShuffleCost + VecCost - ScalarCost;
34623478
}
34633479
case Instruction::FNeg:

llvm/test/Transforms/SLPVectorizer/AArch64/horizontal.ll

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ target triple = "aarch64--linux"
1515
; YAML-NEXT: Function: test_select
1616
; YAML-NEXT: Args:
1717
; YAML-NEXT: - String: 'Vectorized horizontal reduction with cost '
18-
; YAML-NEXT: - Cost: '-8'
18+
; YAML-NEXT: - Cost: '-20'
1919
; YAML-NEXT: - String: ' and with tree size '
2020
; YAML-NEXT: - TreeSize: '8'
2121

@@ -244,7 +244,7 @@ for.end: ; preds = %for.end.loopexit, %
244244
; YAML-NEXT: Function: test_unrolled_select
245245
; YAML-NEXT: Args:
246246
; YAML-NEXT: - String: 'Vectorized horizontal reduction with cost '
247-
; YAML-NEXT: - Cost: '-31'
247+
; YAML-NEXT: - Cost: '-37'
248248
; YAML-NEXT: - String: ' and with tree size '
249249
; YAML-NEXT: - TreeSize: '10'
250250

0 commit comments

Comments
 (0)