Skip to content

[SLP] Extract isIdentityOrder to common routine [probably NFC] #106582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 29, 2024
Merged

[SLP] Extract isIdentityOrder to common routine [probably NFC] #106582

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
387eb6f
[SLP] Extract isIdentityOrder to common routine [probably NFC]
preames Aug 29, 2024
577ce00
Address review comments
preames Aug 29, 2024
23f41a4
Update llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
preames Aug 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 19 additions & 33 deletions llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1364,6 +1364,18 @@ class BoUpSLP {
/// Perform LICM and CSE on the newly generated gather sequences.
void optimizeGatherSequence();

/// Does this non-empty order represent an identity order? Identity
/// should be represented as an empty order, so this is used to
/// decide if we can canonicalize a computed order. Undef elements
Comment on lines +1367 to +1369
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand the description. Why it is empty order?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just describing the behavior of the existing code. We use an empty order to represent no reordering - i.e. the identity order. We use this routine to check if a computed order is identity, and can thus be replaced with an empty one. If you have a preferred wording here, please suggest it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see

/// (represented as size) are ignored.
bool isIdentityOrder(ArrayRef<unsigned> Order) const {
assert(!Order.empty() && "expected non-empty order");
const unsigned Sz = Order.size();
return all_of(enumerate(Order), [&](const auto &P) {
return P.value() == P.index() || P.value() == Sz;
});
}

/// Checks if the specified gather tree entry \p TE can be represented as a
/// shuffled vector entry + (possibly) permutation with other gathers. It
/// implements the checks only for possibly ordered scalars (Loads,
Expand Down Expand Up @@ -5256,12 +5268,6 @@ BoUpSLP::getReorderingData(const TreeEntry &TE, bool TopToBottom) {
}
return I1 < I2;
};
auto IsIdentityOrder = [](const OrdersType &Order) {
for (unsigned Idx : seq<unsigned>(0, Order.size()))
if (Idx != Order[Idx])
return false;
return true;
};
DenseMap<unsigned, unsigned> PhiToId;
SmallVector<unsigned> Phis(TE.Scalars.size());
std::iota(Phis.begin(), Phis.end(), 0);
Expand All @@ -5271,7 +5277,7 @@ BoUpSLP::getReorderingData(const TreeEntry &TE, bool TopToBottom) {
stable_sort(Phis, PHICompare);
for (unsigned Id = 0, Sz = Phis.size(); Id < Sz; ++Id)
ResOrder[Id] = PhiToId[Phis[Id]];
if (IsIdentityOrder(ResOrder))
if (isIdentityOrder(ResOrder))
return std::nullopt; // No need to reorder.
return std::move(ResOrder);
}
Expand Down Expand Up @@ -5565,19 +5571,12 @@ void BoUpSLP::reorderTopToBottom() {
}
if (OrdersUses.empty())
continue;
auto IsIdentityOrder = [](ArrayRef<unsigned> Order) {
const unsigned Sz = Order.size();
for (unsigned Idx : seq<unsigned>(0, Sz))
if (Idx != Order[Idx] && Order[Idx] != Sz)
return false;
return true;
};
// Choose the most used order.
unsigned IdentityCnt = 0;
unsigned FilledIdentityCnt = 0;
OrdersType IdentityOrder(VF, VF);
for (auto &Pair : OrdersUses) {
if (Pair.first.empty() || IsIdentityOrder(Pair.first)) {
if (Pair.first.empty() || isIdentityOrder(Pair.first)) {
if (!Pair.first.empty())
FilledIdentityCnt += Pair.second;
IdentityCnt += Pair.second;
Expand All @@ -5593,7 +5592,7 @@ void BoUpSLP::reorderTopToBottom() {
if (Cnt < Pair.second ||
(Cnt == IdentityCnt && IdentityCnt == FilledIdentityCnt &&
Cnt == Pair.second && !BestOrder.empty() &&
IsIdentityOrder(BestOrder))) {
isIdentityOrder(BestOrder))) {
combineOrders(Pair.first, BestOrder);
BestOrder = Pair.first;
Cnt = Pair.second;
Expand All @@ -5602,7 +5601,7 @@ void BoUpSLP::reorderTopToBottom() {
}
}
// Set order of the user node.
if (IsIdentityOrder(BestOrder))
if (isIdentityOrder(BestOrder))
continue;
fixupOrderingIndices(BestOrder);
SmallVector<int> Mask;
Expand Down Expand Up @@ -5891,19 +5890,12 @@ void BoUpSLP::reorderBottomToTop(bool IgnoreReorder) {
OrderedEntries.remove(Op.second);
continue;
}
auto IsIdentityOrder = [](ArrayRef<unsigned> Order) {
const unsigned Sz = Order.size();
for (unsigned Idx : seq<unsigned>(0, Sz))
if (Idx != Order[Idx] && Order[Idx] != Sz)
return false;
return true;
};
// Choose the most used order.
unsigned IdentityCnt = 0;
unsigned VF = Data.second.front().second->getVectorFactor();
OrdersType IdentityOrder(VF, VF);
for (auto &Pair : OrdersUses) {
if (Pair.first.empty() || IsIdentityOrder(Pair.first)) {
if (Pair.first.empty() || isIdentityOrder(Pair.first)) {
IdentityCnt += Pair.second;
combineOrders(IdentityOrder, Pair.first);
}
Expand All @@ -5923,7 +5915,7 @@ void BoUpSLP::reorderBottomToTop(bool IgnoreReorder) {
}
}
// Set order of the user node.
if (IsIdentityOrder(BestOrder)) {
if (isIdentityOrder(BestOrder)) {
for (const std::pair<unsigned, TreeEntry *> &Op : Data.second)
OrderedEntries.remove(Op.second);
continue;
Expand Down Expand Up @@ -6186,13 +6178,7 @@ bool BoUpSLP::canFormVector(ArrayRef<StoreInst *> StoresVec,
// Identity order (e.g., {0,1,2,3}) is modeled as an empty OrdersType in
// reorderTopToBottom() and reorderBottomToTop(), so we are following the
// same convention here.
auto IsIdentityOrder = [](const OrdersType &Order) {
for (unsigned Idx : seq<unsigned>(0, Order.size()))
if (Idx != Order[Idx])
return false;
return true;
};
if (IsIdentityOrder(ReorderIndices))
if (isIdentityOrder(ReorderIndices))
ReorderIndices.clear();

return true;
Expand Down
Loading