[DAGCombiner] Allow freeze to sink through fmul by adding it to AllowMultipleMaybePoisonOperands #142250
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harrisonGPU
requested review from
jayfoad,
arsenm,
perlfu,
preames,
nhaehnle,
RKSimon,
shiltian and
ruiling
|
@llvm/pr-subscribers-backend-amdgpu @llvm/pr-subscribers-llvm-selectiondag Author: Harrison Hao (harrisonGPU) ChangesAvoid blocking FMA formation on freeze(fmul x, y). Enable combining patterns like:
This improves precision and performance in common numerical code. Closes: #141622 Full diff: https://github.com/llvm/llvm-project/pull/142250.diff 2 Files Affected:
diff --git a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
index be2209a2f8faf..ea5b44da9e48b 100644
--- a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
@@ -16729,6 +16729,28 @@ SDValue DAGCombiner::visitFADDForFMACombine(SDNode *N) {
}
}
+ // fold (fadd (freeze (fmul x, y)), z) -> (fma x, y, z).
+ if ((Options.UnsafeFPMath || N->getFlags().hasAllowContract()) &&
+ N0.getOpcode() == ISD::FREEZE) {
+ SDValue FrozenMul = N0.getOperand(0);
+ if (matcher.match(FrozenMul, ISD::FMUL) && isContractableFMUL(FrozenMul)) {
+ SDValue X = FrozenMul.getOperand(0);
+ SDValue Y = FrozenMul.getOperand(1);
+ return matcher.getNode(PreferredFusedOpcode, SL, VT, X, Y, N1);
+ }
+ }
+
+ // fold (fadd x, (freeze (fmul y, z))) -> (fma y, z, x)
+ if ((Options.UnsafeFPMath || N->getFlags().hasAllowContract()) &&
+ N1.getOpcode() == ISD::FREEZE) {
+ SDValue FrozenMul = N1.getOperand(0);
+ if (matcher.match(FrozenMul, ISD::FMUL) && isContractableFMUL(FrozenMul)) {
+ SDValue X = FrozenMul.getOperand(0);
+ SDValue Y = FrozenMul.getOperand(1);
+ return matcher.getNode(PreferredFusedOpcode, SL, VT, X, Y, N0);
+ }
+ }
+
// More folding opportunities when target permits.
if (Aggressive) {
// fold (fadd (fma x, y, (fpext (fmul u, v))), z)
@@ -17006,6 +17028,30 @@ SDValue DAGCombiner::visitFSUBForFMACombine(SDNode *N) {
}
}
+ // fold (fsub (freeze (fmul x, y)), z) -> (fma x, y, (fneg z))
+ if ((Options.UnsafeFPMath || N->getFlags().hasAllowContract()) &&
+ N0.getOpcode() == ISD::FREEZE) {
+ SDValue FrozenMul = N0.getOperand(0);
+ if (matcher.match(FrozenMul, ISD::FMUL) && isContractableFMUL(FrozenMul)) {
+ SDValue X = FrozenMul.getOperand(0);
+ SDValue Y = FrozenMul.getOperand(1);
+ SDValue NegZ = matcher.getNode(ISD::FNEG, SL, VT, N1);
+ return matcher.getNode(PreferredFusedOpcode, SL, VT, X, Y, NegZ);
+ }
+ }
+
+ // fold (fsub z, (freeze(fmul x, y))) -> (fma (fneg x), y, z)
+ if ((Options.UnsafeFPMath || N->getFlags().hasAllowContract()) &&
+ N1.getOpcode() == ISD::FREEZE) {
+ SDValue FrozenMul = N1.getOperand(0);
+ if (matcher.match(FrozenMul, ISD::FMUL) && isContractableFMUL(FrozenMul)) {
+ SDValue X = FrozenMul.getOperand(0);
+ SDValue Y = FrozenMul.getOperand(1);
+ SDValue NegX = matcher.getNode(ISD::FNEG, SL, VT, X);
+ return matcher.getNode(PreferredFusedOpcode, SL, VT, NegX, Y, N0);
+ }
+ }
+
auto isReassociable = [&Options](SDNode *N) {
return Options.UnsafeFPMath || N->getFlags().hasAllowReassociation();
};
diff --git a/llvm/test/CodeGen/AMDGPU/fold-freeze-fmul-to-fma.ll b/llvm/test/CodeGen/AMDGPU/fold-freeze-fmul-to-fma.ll
new file mode 100644
index 0000000000000..75fe67e743c03
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/fold-freeze-fmul-to-fma.ll
@@ -0,0 +1,54 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1100 -verify-machineinstrs < %s | FileCheck %s -check-prefix GFX11
+
+define float @fma_from_freeze_mul_add_left(float %x, float %y) {
+; GFX11-LABEL: fma_from_freeze_mul_add_left:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT: v_fma_f32 v0, v0, v1, 1.0
+; GFX11-NEXT: s_setpc_b64 s[30:31]
+bb:
+ %mul = fmul contract float %x, %y
+ %mul.fr = freeze float %mul
+ %add = fadd contract float %mul.fr, 1.000000e+00
+ ret float %add
+}
+
+define float @fma_from_freeze_mul_add_right(float %x, float %y) {
+; GFX11-LABEL: fma_from_freeze_mul_add_right:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT: v_fma_f32 v0, v0, v1, 1.0
+; GFX11-NEXT: s_setpc_b64 s[30:31]
+bb:
+ %mul = fmul contract float %x, %y
+ %mul.fr = freeze float %mul
+ %add = fadd contract float 1.000000e+00, %mul.fr
+ ret float %add
+}
+
+define float @fma_from_freeze_mul_sub_left(float %x, float %y) {
+; GFX11-LABEL: fma_from_freeze_mul_sub_left:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT: v_fma_f32 v0, v0, v1, -1.0
+; GFX11-NEXT: s_setpc_b64 s[30:31]
+bb:
+ %mul = fmul contract float %x, %y
+ %mul.fr = freeze float %mul
+ %sub = fsub contract float %mul.fr, 1.000000e+00
+ ret float %sub
+}
+
+define float @fma_from_freeze_mul_sub_right(float %x, float %y) {
+; GFX11-LABEL: fma_from_freeze_mul_sub_right:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX11-NEXT: v_fma_f32 v0, -v0, v1, 1.0
+; GFX11-NEXT: s_setpc_b64 s[30:31]
+bb:
+ %mul = fmul contract float %x, %y
+ %mul.fr = freeze float %mul
+ %sub = fsub contract float 1.000000e+00, %mul.fr
+ ret float %sub
+}
|
|
By the looks of it SelectionDAG::canCreateUndefOrPoison doesn't handle fp ops at all - should we start with fixing those to more closely match ValueTracking llvm::canCreateUndefOrPoison? |
Thanks for the pointer. After rereading the LLVM Language Reference Manual I agree that The LangRef notes for every FP op include:
What do you think of this? References |
|
SelectionDAG::canCreateUndefOrPoison checks for hasPoisonGeneratingFlags - which seems to cover nonans/noinfs - but we'd definitely need to have test coverage for this. |
Thanks a lot! I’ll add a test case with nnan / ninf to verify that canCreateUndefOrPoison correctly returns true when required. |
This was referenced Jun 2, 2025
Merged
harrisonGPU
changed the title
harrisonGPU
changed the title
|
Hi, I want to implement support for folding freeze(fmul) + fadd/fsub into FMA when using the |
|
DAGCombiner::visitFREEZE has a list of "AllowMultipleMaybePoisonOperands" opcodes where it will try to force the freeze through the op and onto all its operandss - I've never been sure of the requirements for this list (or why we don't perform it for all non-poison-generating ops) - but if its permissible to add FMUL to it then that might be a much simpler solution. |
harrisonGPU
changed the title
nikic
approved these changes
Jun 8, 2025
There was a problem hiding this comment.
LGTM
The flag preservation matches the middle-end behavior (https://llvm.godbolt.org/z/a97Yr1Eh5). This assumes that rewrite-based FMF is transparent to freeze.
|
Thanks ! |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/18/builds/17180 Here is the relevant piece of the build log for the reference |
omkar-mohanty
pushed a commit
to omkar-mohanty/llvm-project
that referenced
this pull request
Jun 8, 2025
…MultipleMaybePoisonOperands (llvm#142250) Allow freeze to sink through fmul by treating it as a non-poison-generating op when operands are not poison. Adding `ISD::FMUL` to `AllowMultipleMaybePoisonOperands` lets DAG combine push freeze through fmul. This helps expose patterns like `fmul+fadd` for `FMA` fusion. When rebuilding the node, we drop flags like nnan/ninf/nsz that imply poison, but keep contract, reassoc, afn, and arcp. Closes: llvm#141622
rorth
pushed a commit
to rorth/llvm-project
that referenced
this pull request
Jun 11, 2025
…MultipleMaybePoisonOperands (llvm#142250) Allow freeze to sink through fmul by treating it as a non-poison-generating op when operands are not poison. Adding `ISD::FMUL` to `AllowMultipleMaybePoisonOperands` lets DAG combine push freeze through fmul. This helps expose patterns like `fmul+fadd` for `FMA` fusion. When rebuilding the node, we drop flags like nnan/ninf/nsz that imply poison, but keep contract, reassoc, afn, and arcp. Closes: llvm#141622
DhruvSrivastavaX
pushed a commit
to DhruvSrivastavaX/lldb-for-aix
that referenced
this pull request
Jun 12, 2025
…MultipleMaybePoisonOperands (llvm#142250) Allow freeze to sink through fmul by treating it as a non-poison-generating op when operands are not poison. Adding `ISD::FMUL` to `AllowMultipleMaybePoisonOperands` lets DAG combine push freeze through fmul. This helps expose patterns like `fmul+fadd` for `FMA` fusion. When rebuilding the node, we drop flags like nnan/ninf/nsz that imply poison, but keep contract, reassoc, afn, and arcp. Closes: llvm#141622
tomtor
pushed a commit
to tomtor/llvm-project
that referenced
this pull request
Jun 14, 2025
…MultipleMaybePoisonOperands (llvm#142250) Allow freeze to sink through fmul by treating it as a non-poison-generating op when operands are not poison. Adding `ISD::FMUL` to `AllowMultipleMaybePoisonOperands` lets DAG combine push freeze through fmul. This helps expose patterns like `fmul+fadd` for `FMA` fusion. When rebuilding the node, we drop flags like nnan/ninf/nsz that imply poison, but keep contract, reassoc, afn, and arcp. Closes: llvm#141622
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow freeze to sink through fmul by treating it as a non-poison-generating op
when operands are not poison.
Adding
ISD::FMULtoAllowMultipleMaybePoisonOperandslets DAG combinepush freeze through fmul. This helps expose patterns like
fmul+faddforFMAfusion.When rebuilding the node, we drop flags like nnan/ninf/nsz that imply poison,
but keep contract, reassoc, afn, and arcp.
Closes: #141622