[AMDGPU] Dynamic VGPR support for llvm.amdgcn.cs.chain #130094
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch only adds the instruction for disassembly support. We neither have an instrinsic nor codegen support, and it is unclear whether we actually want to ever have an intrinsic, given the fragile semantics. For now, it will be generated only by the backend in very specific circumstances.
This represents a hardware mode supported only for wave32 compute shaders. When enabled, we set the `.dynamic_vgpr_en` field of `.compute_registers` to true in the PAL metadata.
In dynamic VGPR mode, Waves must deallocate all VGPRs before exiting. If the shader program does not do this, hardware inserts `S_ALLOC_VGPR 0` before S_ENDPGM, but this may incur some performance cost. Therefore it's better if the compiler proactively generates that instruction. This patch extends `si-insert-waitcnts` to deallocate the VGPRs via a `S_ALLOC_VGPR 0` before any `S_ENDPGM` when in dynamic VGPR mode.
The llvm.amdgcn.cs.chain intrinsic has a 'flags' operand which may indicate that we want to reallocate the VGPRs before performing the call. A call with the following arguments: ``` llvm.amdgcn.cs.chain %callee, %exec, %sgpr_args, %vgpr_args, /*flags*/0x1, %num_vgprs, %fallback_exec, %fallback_callee ``` is supposed to do the following: - copy the SGPR and VGPR args into their respective registers - try to change the VGPR allocation - if the allocation has succeeded, set EXEC to %exec and jump to %callee, otherwise set EXEC to %fallback_exec and jump to %fallback_callee This patch implements the dynamic VGPR behaviour by generating an S_ALLOC_VGPR followed by S_CSELECT_B32/64 instructions for the EXEC and callee. The rest of the call sequence is left undisturbed (i.e. identical to the case where the flags are 0 and we don't use dynamic VGPRs). We achieve this by introducing some new pseudos (SI_CS_CHAIN_TC_Wn_DVGPR) which are expanded in the SILateBranchLowering pass, just like the simpler SI_CS_CHAIN_TC_Wn pseudos. The main reason is so that we don't risk other passes (particularly the PostRA scheduler) introducing instructions between the S_ALLOC_VGPR and the jump. Such instructions might end up using VGPRs that have been deallocated, or the wrong EXEC mask. Once the whole backend treats S_ALLOC_VGPR and changes to EXEC as barriers for instructions that use VGPRs, we could in principle move the expansion earlier (but in the absence of a good reason for that my personal preference is to keep it later in order to make debugging easier). Since the expansion happens after register allocation, we're careful to select constants to immediate operands instead of letting ISel generate S_MOVs which could interfere with register allocation (i.e. make it look like we need more registers than we actually do). For GFX12, S_ALLOC_VGPR only works in wave32 mode, so we bail out during ISel in wave64 mode. However, we can define the pseudos for wave64 too so it's easy to handle if future generations support it. Co-authored-by: Ana Mihajlovic <[email protected]>
|
@llvm/pr-subscribers-backend-amdgpu Author: Diana Picus (rovka) ChangesThe llvm.amdgcn.cs.chain intrinsic has a 'flags' operand which may indicate that we want to reallocate the VGPRs before performing the call. A call with the following arguments: is supposed to do the following:
This patch implements the dynamic VGPR behaviour by generating an S_ALLOC_VGPR followed by S_CSELECT_B32/64 instructions for the EXEC and callee. The rest of the call sequence is left undisturbed (i.e. identical to the case where the flags are 0 and we don't use dynamic VGPRs). We achieve this by introducing some new pseudos (SI_CS_CHAIN_TC_Wn_DVGPR) which are expanded in the SILateBranchLowering pass, just like the simpler SI_CS_CHAIN_TC_Wn pseudos. The main reason is so that we don't risk other passes (particularly the PostRA scheduler) introducing instructions between the S_ALLOC_VGPR and the jump. Such instructions might end up using VGPRs that have been deallocated, or the wrong EXEC mask. Once the whole backend treats S_ALLOC_VGPR and changes to EXEC as barriers for instructions that use VGPRs, we could in principle move the expansion earlier (but in the absence of a good reason for that my personal preference is to keep it later in order to make debugging easier). Since the expansion happens after register allocation, we're careful to select constants to immediate operands instead of letting ISel generate S_MOVs which could interfere with register allocation (i.e. make it look like we need more registers than we actually do). For GFX12, S_ALLOC_VGPR only works in wave32 mode, so we bail out during ISel in wave64 mode. However, we can define the pseudos for wave64 too so it's easy to handle if future generations support it. Patch is 94.66 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/130094.diff 11 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGISel.h b/llvm/include/llvm/CodeGen/SelectionDAGISel.h
index e9452a6dc6233..55f8f19d437a0 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGISel.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGISel.h
@@ -328,20 +328,21 @@ class SelectionDAGISel {
};
enum {
- OPFL_None = 0, // Node has no chain or glue input and isn't variadic.
- OPFL_Chain = 1, // Node has a chain input.
- OPFL_GlueInput = 2, // Node has a glue input.
- OPFL_GlueOutput = 4, // Node has a glue output.
- OPFL_MemRefs = 8, // Node gets accumulated MemRefs.
- OPFL_Variadic0 = 1<<4, // Node is variadic, root has 0 fixed inputs.
- OPFL_Variadic1 = 2<<4, // Node is variadic, root has 1 fixed inputs.
- OPFL_Variadic2 = 3<<4, // Node is variadic, root has 2 fixed inputs.
- OPFL_Variadic3 = 4<<4, // Node is variadic, root has 3 fixed inputs.
- OPFL_Variadic4 = 5<<4, // Node is variadic, root has 4 fixed inputs.
- OPFL_Variadic5 = 6<<4, // Node is variadic, root has 5 fixed inputs.
- OPFL_Variadic6 = 7<<4, // Node is variadic, root has 6 fixed inputs.
-
- OPFL_VariadicInfo = OPFL_Variadic6
+ OPFL_None = 0, // Node has no chain or glue input and isn't variadic.
+ OPFL_Chain = 1, // Node has a chain input.
+ OPFL_GlueInput = 2, // Node has a glue input.
+ OPFL_GlueOutput = 4, // Node has a glue output.
+ OPFL_MemRefs = 8, // Node gets accumulated MemRefs.
+ OPFL_Variadic0 = 1 << 4, // Node is variadic, root has 0 fixed inputs.
+ OPFL_Variadic1 = 2 << 4, // Node is variadic, root has 1 fixed inputs.
+ OPFL_Variadic2 = 3 << 4, // Node is variadic, root has 2 fixed inputs.
+ OPFL_Variadic3 = 4 << 4, // Node is variadic, root has 3 fixed inputs.
+ OPFL_Variadic4 = 5 << 4, // Node is variadic, root has 4 fixed inputs.
+ OPFL_Variadic5 = 6 << 4, // Node is variadic, root has 5 fixed inputs.
+ OPFL_Variadic6 = 7 << 4, // Node is variadic, root has 6 fixed inputs.
+ OPFL_Variadic7 = 8 << 4, // Node is variadic, root has 7 fixed inputs.
+
+ OPFL_VariadicInfo = 15 << 4 // Mask for extracting the OPFL_VariadicN bits.
};
/// getNumFixedFromVariadicInfo - Transform an EmitNode flags word into the
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index d5a07e616236e..ba5f7e03d98e4 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -7996,10 +7996,6 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
return;
}
case Intrinsic::amdgcn_cs_chain: {
- assert(I.arg_size() == 5 && "Additional args not supported yet");
- assert(cast<ConstantInt>(I.getOperand(4))->isZero() &&
- "Non-zero flags not supported yet");
-
// At this point we don't care if it's amdgpu_cs_chain or
// amdgpu_cs_chain_preserve.
CallingConv::ID CC = CallingConv::AMDGPU_CS_Chain;
@@ -8026,6 +8022,15 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
assert(!Args[1].IsInReg && "VGPR args should not be marked inreg");
Args[2].IsInReg = true; // EXEC should be inreg
+ // Forward the flags and any additional arguments.
+ for (unsigned Idx = 4; Idx < I.arg_size(); ++Idx) {
+ TargetLowering::ArgListEntry Arg;
+ Arg.Node = getValue(I.getOperand(Idx));
+ Arg.Ty = I.getOperand(Idx)->getType();
+ Arg.setAttributes(&I, Idx);
+ Args.push_back(Arg);
+ }
+
TargetLowering::CallLoweringInfo CLI(DAG);
CLI.setDebugLoc(getCurSDLoc())
.setChain(getRoot())
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp
index 478a4c161fce7..a440617319228 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp
@@ -953,8 +953,9 @@ getAssignFnsForCC(CallingConv::ID CC, const SITargetLowering &TLI) {
}
static unsigned getCallOpcode(const MachineFunction &CallerF, bool IsIndirect,
- bool IsTailCall, bool isWave32,
- CallingConv::ID CC) {
+ bool IsTailCall, bool IsWave32,
+ CallingConv::ID CC,
+ bool IsDynamicVGPRChainCall = false) {
// For calls to amdgpu_cs_chain functions, the address is known to be uniform.
assert((AMDGPU::isChainCC(CC) || !IsIndirect || !IsTailCall) &&
"Indirect calls can't be tail calls, "
@@ -962,8 +963,12 @@ static unsigned getCallOpcode(const MachineFunction &CallerF, bool IsIndirect,
if (!IsTailCall)
return AMDGPU::G_SI_CALL;
- if (AMDGPU::isChainCC(CC))
- return isWave32 ? AMDGPU::SI_CS_CHAIN_TC_W32 : AMDGPU::SI_CS_CHAIN_TC_W64;
+ if (AMDGPU::isChainCC(CC)) {
+ if (IsDynamicVGPRChainCall)
+ return IsWave32 ? AMDGPU::SI_CS_CHAIN_TC_W32_DVGPR
+ : AMDGPU::SI_CS_CHAIN_TC_W64_DVGPR;
+ return IsWave32 ? AMDGPU::SI_CS_CHAIN_TC_W32 : AMDGPU::SI_CS_CHAIN_TC_W64;
+ }
return CC == CallingConv::AMDGPU_Gfx ? AMDGPU::SI_TCRETURN_GFX :
AMDGPU::SI_TCRETURN;
@@ -972,7 +977,8 @@ static unsigned getCallOpcode(const MachineFunction &CallerF, bool IsIndirect,
// Add operands to call instruction to track the callee.
static bool addCallTargetOperands(MachineInstrBuilder &CallInst,
MachineIRBuilder &MIRBuilder,
- AMDGPUCallLowering::CallLoweringInfo &Info) {
+ AMDGPUCallLowering::CallLoweringInfo &Info,
+ bool IsDynamicVGPRChainCall = false) {
if (Info.Callee.isReg()) {
CallInst.addReg(Info.Callee.getReg());
CallInst.addImm(0);
@@ -983,7 +989,12 @@ static bool addCallTargetOperands(MachineInstrBuilder &CallInst,
auto Ptr = MIRBuilder.buildGlobalValue(
LLT::pointer(GV->getAddressSpace(), 64), GV);
CallInst.addReg(Ptr.getReg(0));
- CallInst.add(Info.Callee);
+
+ if (IsDynamicVGPRChainCall)
+ // DynamicVGPR chain calls are always indirect.
+ CallInst.addImm(0);
+ else
+ CallInst.add(Info.Callee);
} else
return false;
@@ -1177,6 +1188,18 @@ void AMDGPUCallLowering::handleImplicitCallArguments(
}
}
+namespace {
+// Chain calls have special arguments that we need to handle. These have the
+// same index as they do in the llvm.amdgcn.cs.chain intrinsic.
+enum ChainCallArgIdx {
+ Exec = 1,
+ Flags = 4,
+ NumVGPRs = 5,
+ FallbackExec = 6,
+ FallbackCallee = 7,
+};
+} // anonymous namespace
+
bool AMDGPUCallLowering::lowerTailCall(
MachineIRBuilder &MIRBuilder, CallLoweringInfo &Info,
SmallVectorImpl<ArgInfo> &OutArgs) const {
@@ -1185,6 +1208,8 @@ bool AMDGPUCallLowering::lowerTailCall(
SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
const Function &F = MF.getFunction();
MachineRegisterInfo &MRI = MF.getRegInfo();
+ const SIInstrInfo *TII = ST.getInstrInfo();
+ const SIRegisterInfo *TRI = ST.getRegisterInfo();
const SITargetLowering &TLI = *getTLI<SITargetLowering>();
// True when we're tail calling, but without -tailcallopt.
@@ -1200,34 +1225,78 @@ bool AMDGPUCallLowering::lowerTailCall(
if (!IsSibCall)
CallSeqStart = MIRBuilder.buildInstr(AMDGPU::ADJCALLSTACKUP);
- unsigned Opc =
- getCallOpcode(MF, Info.Callee.isReg(), true, ST.isWave32(), CalleeCC);
+ bool IsChainCall = AMDGPU::isChainCC(Info.CallConv);
+ bool IsDynamicVGPRChainCall = false;
+
+ if (IsChainCall) {
+ ArgInfo FlagsArg = Info.OrigArgs[ChainCallArgIdx::Flags];
+ const APInt &FlagsValue = cast<ConstantInt>(FlagsArg.OrigValue)->getValue();
+ if (FlagsValue.isZero()) {
+ if (Info.OrigArgs.size() != 5) {
+ LLVM_DEBUG(dbgs() << "No additional args allowed if flags == 0");
+ return false;
+ }
+ } else if (FlagsValue.isOneBitSet(0)) {
+ IsDynamicVGPRChainCall = true;
+
+ if (Info.OrigArgs.size() != 8) {
+ LLVM_DEBUG(dbgs() << "Expected 3 additional args");
+ return false;
+ }
+
+ // On GFX12, we can only change the VGPR allocation for wave32.
+ if (!ST.isWave32()) {
+ LLVM_DEBUG(dbgs() << "Dynamic VGPR mode is only supported for wave32");
+ return false;
+ }
+
+ ArgInfo FallbackExecArg = Info.OrigArgs[ChainCallArgIdx::FallbackExec];
+ assert(FallbackExecArg.Regs.size() == 1 &&
+ "Expected single register for fallback EXEC");
+ if (!FallbackExecArg.Ty->isIntegerTy(ST.getWavefrontSize())) {
+ LLVM_DEBUG(dbgs() << "Bad type for fallback EXEC");
+ return false;
+ }
+ }
+ }
+
+ unsigned Opc = getCallOpcode(MF, Info.Callee.isReg(), /*IsTailCall*/ true,
+ ST.isWave32(), CalleeCC, IsDynamicVGPRChainCall);
auto MIB = MIRBuilder.buildInstrNoInsert(Opc);
- if (!addCallTargetOperands(MIB, MIRBuilder, Info))
+ if (!addCallTargetOperands(MIB, MIRBuilder, Info, IsDynamicVGPRChainCall))
return false;
// Byte offset for the tail call. When we are sibcalling, this will always
// be 0.
MIB.addImm(0);
- // If this is a chain call, we need to pass in the EXEC mask.
- const SIRegisterInfo *TRI = ST.getRegisterInfo();
- if (AMDGPU::isChainCC(Info.CallConv)) {
- ArgInfo ExecArg = Info.OrigArgs[1];
+ // If this is a chain call, we need to pass in the EXEC mask as well as any
+ // other special args.
+ if (IsChainCall) {
+ auto AddRegOrImm = [&](const ArgInfo &Arg) {
+ if (auto CI = dyn_cast<ConstantInt>(Arg.OrigValue)) {
+ MIB.addImm(CI->getSExtValue());
+ } else {
+ MIB.addReg(Arg.Regs[0]);
+ unsigned Idx = MIB->getNumOperands() - 1;
+ MIB->getOperand(Idx).setReg(constrainOperandRegClass(
+ MF, *TRI, MRI, *TII, *ST.getRegBankInfo(), *MIB, MIB->getDesc(),
+ MIB->getOperand(Idx), Idx));
+ }
+ };
+
+ ArgInfo ExecArg = Info.OrigArgs[ChainCallArgIdx::Exec];
assert(ExecArg.Regs.size() == 1 && "Too many regs for EXEC");
- if (!ExecArg.Ty->isIntegerTy(ST.getWavefrontSize()))
+ if (!ExecArg.Ty->isIntegerTy(ST.getWavefrontSize())) {
+ LLVM_DEBUG(dbgs() << "Bad type for EXEC");
return false;
-
- if (const auto *CI = dyn_cast<ConstantInt>(ExecArg.OrigValue)) {
- MIB.addImm(CI->getSExtValue());
- } else {
- MIB.addReg(ExecArg.Regs[0]);
- unsigned Idx = MIB->getNumOperands() - 1;
- MIB->getOperand(Idx).setReg(constrainOperandRegClass(
- MF, *TRI, MRI, *ST.getInstrInfo(), *ST.getRegBankInfo(), *MIB,
- MIB->getDesc(), MIB->getOperand(Idx), Idx));
}
+
+ AddRegOrImm(ExecArg);
+ if (IsDynamicVGPRChainCall)
+ std::for_each(Info.OrigArgs.begin() + ChainCallArgIdx::NumVGPRs,
+ Info.OrigArgs.end(), AddRegOrImm);
}
// Tell the call which registers are clobbered.
@@ -1329,9 +1398,9 @@ bool AMDGPUCallLowering::lowerTailCall(
// FIXME: We should define regbankselectable call instructions to handle
// divergent call targets.
if (MIB->getOperand(0).isReg()) {
- MIB->getOperand(0).setReg(constrainOperandRegClass(
- MF, *TRI, MRI, *ST.getInstrInfo(), *ST.getRegBankInfo(), *MIB,
- MIB->getDesc(), MIB->getOperand(0), 0));
+ MIB->getOperand(0).setReg(
+ constrainOperandRegClass(MF, *TRI, MRI, *TII, *ST.getRegBankInfo(),
+ *MIB, MIB->getDesc(), MIB->getOperand(0), 0));
}
MF.getFrameInfo().setHasTailCall();
@@ -1345,11 +1414,6 @@ bool AMDGPUCallLowering::lowerChainCall(MachineIRBuilder &MIRBuilder,
ArgInfo Callee = Info.OrigArgs[0];
ArgInfo SGPRArgs = Info.OrigArgs[2];
ArgInfo VGPRArgs = Info.OrigArgs[3];
- ArgInfo Flags = Info.OrigArgs[4];
-
- assert(cast<ConstantInt>(Flags.OrigValue)->isZero() &&
- "Non-zero flags aren't supported yet.");
- assert(Info.OrigArgs.size() == 5 && "Additional args aren't supported yet.");
MachineFunction &MF = MIRBuilder.getMF();
const Function &F = MF.getFunction();
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index fe095414e5172..5438c6f50dea2 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -3657,6 +3657,19 @@ bool SITargetLowering::mayBeEmittedAsTailCall(const CallInst *CI) const {
return true;
}
+namespace {
+// Chain calls have special arguments that we need to handle. These are
+// tagging along at the end of the arguments list(s), after the SGPR and VGPR
+// arguments (index 0 and 1 respectively).
+enum ChainCallArgIdx {
+ Exec = 2,
+ Flags,
+ NumVGPRs,
+ FallbackExec,
+ FallbackCallee
+};
+} // anonymous namespace
+
// The wave scratch offset register is used as the global base pointer.
SDValue SITargetLowering::LowerCall(CallLoweringInfo &CLI,
SmallVectorImpl<SDValue> &InVals) const {
@@ -3665,37 +3678,67 @@ SDValue SITargetLowering::LowerCall(CallLoweringInfo &CLI,
SelectionDAG &DAG = CLI.DAG;
- TargetLowering::ArgListEntry RequestedExec;
- if (IsChainCallConv) {
- // The last argument should be the value that we need to put in EXEC.
- // Pop it out of CLI.Outs and CLI.OutVals before we do any processing so we
- // don't treat it like the rest of the arguments.
- RequestedExec = CLI.Args.back();
- assert(RequestedExec.Node && "No node for EXEC");
+ const SDLoc &DL = CLI.DL;
+ SDValue Chain = CLI.Chain;
+ SDValue Callee = CLI.Callee;
- if (!RequestedExec.Ty->isIntegerTy(Subtarget->getWavefrontSize()))
+ llvm::SmallVector<SDValue, 6> ChainCallSpecialArgs;
+ if (IsChainCallConv) {
+ // The last arguments should be the value that we need to put in EXEC,
+ // followed by the flags and any other arguments with special meanings.
+ // Pop them out of CLI.Outs and CLI.OutVals before we do any processing so
+ // we don't treat them like the "real" arguments.
+ auto RequestedExecIt = std::find_if(
+ CLI.Outs.begin(), CLI.Outs.end(),
+ [](const ISD::OutputArg &Arg) { return Arg.OrigArgIndex == 2; });
+ assert(RequestedExecIt != CLI.Outs.end() && "No node for EXEC");
+
+ size_t SpecialArgsBeginIdx = RequestedExecIt - CLI.Outs.begin();
+ CLI.OutVals.erase(CLI.OutVals.begin() + SpecialArgsBeginIdx,
+ CLI.OutVals.end());
+ CLI.Outs.erase(RequestedExecIt, CLI.Outs.end());
+
+ assert(CLI.Outs.back().OrigArgIndex < 2 &&
+ "Haven't popped all the special args");
+
+ TargetLowering::ArgListEntry RequestedExecArg =
+ CLI.Args[ChainCallArgIdx::Exec];
+ if (!RequestedExecArg.Ty->isIntegerTy(Subtarget->getWavefrontSize()))
return lowerUnhandledCall(CLI, InVals, "Invalid value for EXEC");
- assert(CLI.Outs.back().OrigArgIndex == 2 && "Unexpected last arg");
- CLI.Outs.pop_back();
- CLI.OutVals.pop_back();
+ // Convert constants into TargetConstants, so they become immediate operands
+ // instead of being selected into S_MOV.
+ auto PushNodeOrTargetConstant = [&](TargetLowering::ArgListEntry Arg) {
+ if (auto ArgNode = dyn_cast<ConstantSDNode>(Arg.Node))
+ ChainCallSpecialArgs.push_back(DAG.getTargetConstant(
+ ArgNode->getAPIntValue(), DL, ArgNode->getValueType(0)));
+ else
+ ChainCallSpecialArgs.push_back(Arg.Node);
+ };
- if (RequestedExec.Ty->isIntegerTy(64)) {
- assert(CLI.Outs.back().OrigArgIndex == 2 && "Exec wasn't split up");
- CLI.Outs.pop_back();
- CLI.OutVals.pop_back();
- }
+ PushNodeOrTargetConstant(RequestedExecArg);
+
+ // Process any other special arguments depending on the value of the flags.
+ TargetLowering::ArgListEntry Flags = CLI.Args[ChainCallArgIdx::Flags];
+
+ const APInt &FlagsValue = cast<ConstantSDNode>(Flags.Node)->getAPIntValue();
+ if (FlagsValue.isZero()) {
+ if (CLI.Args.size() > ChainCallArgIdx::Flags + 1)
+ return lowerUnhandledCall(CLI, InVals,
+ "No additional args allowed if flags == 0");
+ } else if (FlagsValue.isOneBitSet(0)) {
+ if (CLI.Args.size() != ChainCallArgIdx::FallbackCallee + 1) {
+ return lowerUnhandledCall(CLI, InVals, "Expected 3 additional args");
+ }
- assert(CLI.Outs.back().OrigArgIndex != 2 &&
- "Haven't popped all the pieces of the EXEC mask");
+ std::for_each(CLI.Args.begin() + ChainCallArgIdx::NumVGPRs,
+ CLI.Args.end(), PushNodeOrTargetConstant);
+ }
}
- const SDLoc &DL = CLI.DL;
SmallVector<ISD::OutputArg, 32> &Outs = CLI.Outs;
SmallVector<SDValue, 32> &OutVals = CLI.OutVals;
SmallVector<ISD::InputArg, 32> &Ins = CLI.Ins;
- SDValue Chain = CLI.Chain;
- SDValue Callee = CLI.Callee;
bool &IsTailCall = CLI.IsTailCall;
bool IsVarArg = CLI.IsVarArg;
bool IsSibCall = false;
@@ -3983,7 +4026,8 @@ SDValue SITargetLowering::LowerCall(CallLoweringInfo &CLI,
}
if (IsChainCallConv)
- Ops.push_back(RequestedExec.Node);
+ Ops.insert(Ops.end(), ChainCallSpecialArgs.begin(),
+ ChainCallSpecialArgs.end());
// Add argument registers to the end of the list so that they are known live
// into the call.
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 63f66023837a2..e5af35f2da686 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -692,36 +692,42 @@ def : GCNPat<
(SI_TCRETURN_GFX Gfx_CCR_SGPR_64:$src0, (i64 0), i32imm:$fpdiff)
>;
-// Pseudo for the llvm.amdgcn.cs.chain intrinsic.
-// This is essentially a tail call, but it also takes a mask to put in EXEC
-// right before jumping to the callee.
-class SI_CS_CHAIN_TC<
+// Pseudos for the llvm.amdgcn.cs.chain intrinsic.
+multiclass SI_CS_CHAIN_TC<
ValueType execvt, Predicate wavesizepred,
- RegisterOperand execrc = getSOPSrcForVT<execvt>.ret>
- : SPseudoInstSI <(outs),
- (ins CCR_SGPR_64:$src0, unknown:$callee, i32imm:$fpdiff, execrc:$exec)> {
- let FixedSize = 0;
- let isCall = 1;
- let isTerminator = 1;
- let isBarrier = 1;
- let isReturn = 1;
- let UseNamedOperandTable = 1;
- let SchedRW = [WriteBranch];
- let isConvergent = 1;
-
- let WaveSizePredicate = wavesizepred;
-}
-
-def SI_CS_CHAIN_TC_W32 : SI_CS_CHAIN_TC<i32, isWave32>;
-def SI_CS_CHAIN_TC_W64 : SI_CS_CHAIN_TC<i64, isWave64>;
+ RegisterOperand execrc = getSOPSrcForVT<execvt>.ret> {
+ let FixedSize = 0,
+ isCall = 1,
+ isTerminator = 1,
+ isBarrier = 1,
+ isReturn = 1,
+ UseNamedOperandTable = 1,
+ SchedRW = [WriteBranch],
+ isConvergent = 1,
+ WaveSizePredicate = wavesizepred in {
+ // This is essentially a tail call, but it also takes a mask to put in EXEC
+ // right before jumping to the callee.
+ def NAME: SPseudoInstSI <(outs),
+ (ins CCR_SGPR_64:$src0, unknown:$callee, i32imm:$fpdiff, execrc:$exec)>;
+
+ // Same as above, but it will first try to reallocate the VGPRs, and choose an
+ // EXEC mask and a callee depending on the success of the reallocation attempt.
+ def _DVGPR : SPseudoInstSI <(outs),
+ (ins CCR_SGPR_64:$src0, i64imm:$callee, i32imm:$fpdiff, execrc:$exec,
+ SSrc_b32:$numvgprs, execrc:$fbexec, CCR_SGPR_64:$fbcallee)>;
+ } // End FixedSize = 0 etc
+}
+
+defm SI_CS_CHAIN_TC_W32 : SI_CS_CHAIN_TC<i32, isWave32>;
+defm SI_CS_CHAIN_TC_W64 : SI_CS_CHAIN_TC<i64, isWave64>;
// Handle selecting direct & indirect calls via SI_CS_CHAIN_TC_W32/64
multiclass si_cs_chain_tc_pattern<
dag callee, ValueType execvt, RegisterOperand execrc, Instruction tc> {
-def : GCNPat<
- (AMDGPUtc_return_chain i64:$src0, callee, (i32 timm:$fpdiff), execvt:$exec),
- (tc CCR_SGPR_64:$src0, callee, i32imm:$fpdiff, execrc:$exec)
->;
+ def : GCNPat<
+ (AMDGPUtc_return_chain i64:$src0, callee, (i32 timm:$fpdiff), execvt:$exec),
+ (tc CCR_SGPR_64:$src0, callee, i32imm:$fpdiff, execrc:$exec)
+ >;
}
multiclass si_cs_chain_tc_patterns<
@@ -736,6 +742,26 @@ multiclass si_cs_chain_tc_patterns<
defm : si_cs_chain_tc_patterns<i32>;
defm : si_cs_chain_tc_patterns<i64>;
+// Match dynamic VGPR case. This is always indirect since we choose the callee
+// dynamically based on the result of the VGPR reall...
[truncated]
|
|
@llvm/pr-subscribers-llvm-selectiondag Author: Diana Picus (rovka) ChangesThe llvm.amdgcn.cs.chain intrinsic has a 'flags' operand which may indicate that we want to reallocate the VGPRs before performing the call. A call with the following arguments: is supposed to do the following:
This patch implements the dynamic VGPR behaviour by generating an S_ALLOC_VGPR followed by S_CSELECT_B32/64 instructions for the EXEC and callee. The rest of the call sequence is left undisturbed (i.e. identical to the case where the flags are 0 and we don't use dynamic VGPRs). We achieve this by introducing some new pseudos (SI_CS_CHAIN_TC_Wn_DVGPR) which are expanded in the SILateBranchLowering pass, just like the simpler SI_CS_CHAIN_TC_Wn pseudos. The main reason is so that we don't risk other passes (particularly the PostRA scheduler) introducing instructions between the S_ALLOC_VGPR and the jump. Such instructions might end up using VGPRs that have been deallocated, or the wrong EXEC mask. Once the whole backend treats S_ALLOC_VGPR and changes to EXEC as barriers for instructions that use VGPRs, we could in principle move the expansion earlier (but in the absence of a good reason for that my personal preference is to keep it later in order to make debugging easier). Since the expansion happens after register allocation, we're careful to select constants to immediate operands instead of letting ISel generate S_MOVs which could interfere with register allocation (i.e. make it look like we need more registers than we actually do). For GFX12, S_ALLOC_VGPR only works in wave32 mode, so we bail out during ISel in wave64 mode. However, we can define the pseudos for wave64 too so it's easy to handle if future generations support it. Patch is 94.66 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/130094.diff 11 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGISel.h b/llvm/include/llvm/CodeGen/SelectionDAGISel.h
index e9452a6dc6233..55f8f19d437a0 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGISel.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGISel.h
@@ -328,20 +328,21 @@ class SelectionDAGISel {
};
enum {
- OPFL_None = 0, // Node has no chain or glue input and isn't variadic.
- OPFL_Chain = 1, // Node has a chain input.
- OPFL_GlueInput = 2, // Node has a glue input.
- OPFL_GlueOutput = 4, // Node has a glue output.
- OPFL_MemRefs = 8, // Node gets accumulated MemRefs.
- OPFL_Variadic0 = 1<<4, // Node is variadic, root has 0 fixed inputs.
- OPFL_Variadic1 = 2<<4, // Node is variadic, root has 1 fixed inputs.
- OPFL_Variadic2 = 3<<4, // Node is variadic, root has 2 fixed inputs.
- OPFL_Variadic3 = 4<<4, // Node is variadic, root has 3 fixed inputs.
- OPFL_Variadic4 = 5<<4, // Node is variadic, root has 4 fixed inputs.
- OPFL_Variadic5 = 6<<4, // Node is variadic, root has 5 fixed inputs.
- OPFL_Variadic6 = 7<<4, // Node is variadic, root has 6 fixed inputs.
-
- OPFL_VariadicInfo = OPFL_Variadic6
+ OPFL_None = 0, // Node has no chain or glue input and isn't variadic.
+ OPFL_Chain = 1, // Node has a chain input.
+ OPFL_GlueInput = 2, // Node has a glue input.
+ OPFL_GlueOutput = 4, // Node has a glue output.
+ OPFL_MemRefs = 8, // Node gets accumulated MemRefs.
+ OPFL_Variadic0 = 1 << 4, // Node is variadic, root has 0 fixed inputs.
+ OPFL_Variadic1 = 2 << 4, // Node is variadic, root has 1 fixed inputs.
+ OPFL_Variadic2 = 3 << 4, // Node is variadic, root has 2 fixed inputs.
+ OPFL_Variadic3 = 4 << 4, // Node is variadic, root has 3 fixed inputs.
+ OPFL_Variadic4 = 5 << 4, // Node is variadic, root has 4 fixed inputs.
+ OPFL_Variadic5 = 6 << 4, // Node is variadic, root has 5 fixed inputs.
+ OPFL_Variadic6 = 7 << 4, // Node is variadic, root has 6 fixed inputs.
+ OPFL_Variadic7 = 8 << 4, // Node is variadic, root has 7 fixed inputs.
+
+ OPFL_VariadicInfo = 15 << 4 // Mask for extracting the OPFL_VariadicN bits.
};
/// getNumFixedFromVariadicInfo - Transform an EmitNode flags word into the
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index d5a07e616236e..ba5f7e03d98e4 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -7996,10 +7996,6 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
return;
}
case Intrinsic::amdgcn_cs_chain: {
- assert(I.arg_size() == 5 && "Additional args not supported yet");
- assert(cast<ConstantInt>(I.getOperand(4))->isZero() &&
- "Non-zero flags not supported yet");
-
// At this point we don't care if it's amdgpu_cs_chain or
// amdgpu_cs_chain_preserve.
CallingConv::ID CC = CallingConv::AMDGPU_CS_Chain;
@@ -8026,6 +8022,15 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
assert(!Args[1].IsInReg && "VGPR args should not be marked inreg");
Args[2].IsInReg = true; // EXEC should be inreg
+ // Forward the flags and any additional arguments.
+ for (unsigned Idx = 4; Idx < I.arg_size(); ++Idx) {
+ TargetLowering::ArgListEntry Arg;
+ Arg.Node = getValue(I.getOperand(Idx));
+ Arg.Ty = I.getOperand(Idx)->getType();
+ Arg.setAttributes(&I, Idx);
+ Args.push_back(Arg);
+ }
+
TargetLowering::CallLoweringInfo CLI(DAG);
CLI.setDebugLoc(getCurSDLoc())
.setChain(getRoot())
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp
index 478a4c161fce7..a440617319228 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp
@@ -953,8 +953,9 @@ getAssignFnsForCC(CallingConv::ID CC, const SITargetLowering &TLI) {
}
static unsigned getCallOpcode(const MachineFunction &CallerF, bool IsIndirect,
- bool IsTailCall, bool isWave32,
- CallingConv::ID CC) {
+ bool IsTailCall, bool IsWave32,
+ CallingConv::ID CC,
+ bool IsDynamicVGPRChainCall = false) {
// For calls to amdgpu_cs_chain functions, the address is known to be uniform.
assert((AMDGPU::isChainCC(CC) || !IsIndirect || !IsTailCall) &&
"Indirect calls can't be tail calls, "
@@ -962,8 +963,12 @@ static unsigned getCallOpcode(const MachineFunction &CallerF, bool IsIndirect,
if (!IsTailCall)
return AMDGPU::G_SI_CALL;
- if (AMDGPU::isChainCC(CC))
- return isWave32 ? AMDGPU::SI_CS_CHAIN_TC_W32 : AMDGPU::SI_CS_CHAIN_TC_W64;
+ if (AMDGPU::isChainCC(CC)) {
+ if (IsDynamicVGPRChainCall)
+ return IsWave32 ? AMDGPU::SI_CS_CHAIN_TC_W32_DVGPR
+ : AMDGPU::SI_CS_CHAIN_TC_W64_DVGPR;
+ return IsWave32 ? AMDGPU::SI_CS_CHAIN_TC_W32 : AMDGPU::SI_CS_CHAIN_TC_W64;
+ }
return CC == CallingConv::AMDGPU_Gfx ? AMDGPU::SI_TCRETURN_GFX :
AMDGPU::SI_TCRETURN;
@@ -972,7 +977,8 @@ static unsigned getCallOpcode(const MachineFunction &CallerF, bool IsIndirect,
// Add operands to call instruction to track the callee.
static bool addCallTargetOperands(MachineInstrBuilder &CallInst,
MachineIRBuilder &MIRBuilder,
- AMDGPUCallLowering::CallLoweringInfo &Info) {
+ AMDGPUCallLowering::CallLoweringInfo &Info,
+ bool IsDynamicVGPRChainCall = false) {
if (Info.Callee.isReg()) {
CallInst.addReg(Info.Callee.getReg());
CallInst.addImm(0);
@@ -983,7 +989,12 @@ static bool addCallTargetOperands(MachineInstrBuilder &CallInst,
auto Ptr = MIRBuilder.buildGlobalValue(
LLT::pointer(GV->getAddressSpace(), 64), GV);
CallInst.addReg(Ptr.getReg(0));
- CallInst.add(Info.Callee);
+
+ if (IsDynamicVGPRChainCall)
+ // DynamicVGPR chain calls are always indirect.
+ CallInst.addImm(0);
+ else
+ CallInst.add(Info.Callee);
} else
return false;
@@ -1177,6 +1188,18 @@ void AMDGPUCallLowering::handleImplicitCallArguments(
}
}
+namespace {
+// Chain calls have special arguments that we need to handle. These have the
+// same index as they do in the llvm.amdgcn.cs.chain intrinsic.
+enum ChainCallArgIdx {
+ Exec = 1,
+ Flags = 4,
+ NumVGPRs = 5,
+ FallbackExec = 6,
+ FallbackCallee = 7,
+};
+} // anonymous namespace
+
bool AMDGPUCallLowering::lowerTailCall(
MachineIRBuilder &MIRBuilder, CallLoweringInfo &Info,
SmallVectorImpl<ArgInfo> &OutArgs) const {
@@ -1185,6 +1208,8 @@ bool AMDGPUCallLowering::lowerTailCall(
SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
const Function &F = MF.getFunction();
MachineRegisterInfo &MRI = MF.getRegInfo();
+ const SIInstrInfo *TII = ST.getInstrInfo();
+ const SIRegisterInfo *TRI = ST.getRegisterInfo();
const SITargetLowering &TLI = *getTLI<SITargetLowering>();
// True when we're tail calling, but without -tailcallopt.
@@ -1200,34 +1225,78 @@ bool AMDGPUCallLowering::lowerTailCall(
if (!IsSibCall)
CallSeqStart = MIRBuilder.buildInstr(AMDGPU::ADJCALLSTACKUP);
- unsigned Opc =
- getCallOpcode(MF, Info.Callee.isReg(), true, ST.isWave32(), CalleeCC);
+ bool IsChainCall = AMDGPU::isChainCC(Info.CallConv);
+ bool IsDynamicVGPRChainCall = false;
+
+ if (IsChainCall) {
+ ArgInfo FlagsArg = Info.OrigArgs[ChainCallArgIdx::Flags];
+ const APInt &FlagsValue = cast<ConstantInt>(FlagsArg.OrigValue)->getValue();
+ if (FlagsValue.isZero()) {
+ if (Info.OrigArgs.size() != 5) {
+ LLVM_DEBUG(dbgs() << "No additional args allowed if flags == 0");
+ return false;
+ }
+ } else if (FlagsValue.isOneBitSet(0)) {
+ IsDynamicVGPRChainCall = true;
+
+ if (Info.OrigArgs.size() != 8) {
+ LLVM_DEBUG(dbgs() << "Expected 3 additional args");
+ return false;
+ }
+
+ // On GFX12, we can only change the VGPR allocation for wave32.
+ if (!ST.isWave32()) {
+ LLVM_DEBUG(dbgs() << "Dynamic VGPR mode is only supported for wave32");
+ return false;
+ }
+
+ ArgInfo FallbackExecArg = Info.OrigArgs[ChainCallArgIdx::FallbackExec];
+ assert(FallbackExecArg.Regs.size() == 1 &&
+ "Expected single register for fallback EXEC");
+ if (!FallbackExecArg.Ty->isIntegerTy(ST.getWavefrontSize())) {
+ LLVM_DEBUG(dbgs() << "Bad type for fallback EXEC");
+ return false;
+ }
+ }
+ }
+
+ unsigned Opc = getCallOpcode(MF, Info.Callee.isReg(), /*IsTailCall*/ true,
+ ST.isWave32(), CalleeCC, IsDynamicVGPRChainCall);
auto MIB = MIRBuilder.buildInstrNoInsert(Opc);
- if (!addCallTargetOperands(MIB, MIRBuilder, Info))
+ if (!addCallTargetOperands(MIB, MIRBuilder, Info, IsDynamicVGPRChainCall))
return false;
// Byte offset for the tail call. When we are sibcalling, this will always
// be 0.
MIB.addImm(0);
- // If this is a chain call, we need to pass in the EXEC mask.
- const SIRegisterInfo *TRI = ST.getRegisterInfo();
- if (AMDGPU::isChainCC(Info.CallConv)) {
- ArgInfo ExecArg = Info.OrigArgs[1];
+ // If this is a chain call, we need to pass in the EXEC mask as well as any
+ // other special args.
+ if (IsChainCall) {
+ auto AddRegOrImm = [&](const ArgInfo &Arg) {
+ if (auto CI = dyn_cast<ConstantInt>(Arg.OrigValue)) {
+ MIB.addImm(CI->getSExtValue());
+ } else {
+ MIB.addReg(Arg.Regs[0]);
+ unsigned Idx = MIB->getNumOperands() - 1;
+ MIB->getOperand(Idx).setReg(constrainOperandRegClass(
+ MF, *TRI, MRI, *TII, *ST.getRegBankInfo(), *MIB, MIB->getDesc(),
+ MIB->getOperand(Idx), Idx));
+ }
+ };
+
+ ArgInfo ExecArg = Info.OrigArgs[ChainCallArgIdx::Exec];
assert(ExecArg.Regs.size() == 1 && "Too many regs for EXEC");
- if (!ExecArg.Ty->isIntegerTy(ST.getWavefrontSize()))
+ if (!ExecArg.Ty->isIntegerTy(ST.getWavefrontSize())) {
+ LLVM_DEBUG(dbgs() << "Bad type for EXEC");
return false;
-
- if (const auto *CI = dyn_cast<ConstantInt>(ExecArg.OrigValue)) {
- MIB.addImm(CI->getSExtValue());
- } else {
- MIB.addReg(ExecArg.Regs[0]);
- unsigned Idx = MIB->getNumOperands() - 1;
- MIB->getOperand(Idx).setReg(constrainOperandRegClass(
- MF, *TRI, MRI, *ST.getInstrInfo(), *ST.getRegBankInfo(), *MIB,
- MIB->getDesc(), MIB->getOperand(Idx), Idx));
}
+
+ AddRegOrImm(ExecArg);
+ if (IsDynamicVGPRChainCall)
+ std::for_each(Info.OrigArgs.begin() + ChainCallArgIdx::NumVGPRs,
+ Info.OrigArgs.end(), AddRegOrImm);
}
// Tell the call which registers are clobbered.
@@ -1329,9 +1398,9 @@ bool AMDGPUCallLowering::lowerTailCall(
// FIXME: We should define regbankselectable call instructions to handle
// divergent call targets.
if (MIB->getOperand(0).isReg()) {
- MIB->getOperand(0).setReg(constrainOperandRegClass(
- MF, *TRI, MRI, *ST.getInstrInfo(), *ST.getRegBankInfo(), *MIB,
- MIB->getDesc(), MIB->getOperand(0), 0));
+ MIB->getOperand(0).setReg(
+ constrainOperandRegClass(MF, *TRI, MRI, *TII, *ST.getRegBankInfo(),
+ *MIB, MIB->getDesc(), MIB->getOperand(0), 0));
}
MF.getFrameInfo().setHasTailCall();
@@ -1345,11 +1414,6 @@ bool AMDGPUCallLowering::lowerChainCall(MachineIRBuilder &MIRBuilder,
ArgInfo Callee = Info.OrigArgs[0];
ArgInfo SGPRArgs = Info.OrigArgs[2];
ArgInfo VGPRArgs = Info.OrigArgs[3];
- ArgInfo Flags = Info.OrigArgs[4];
-
- assert(cast<ConstantInt>(Flags.OrigValue)->isZero() &&
- "Non-zero flags aren't supported yet.");
- assert(Info.OrigArgs.size() == 5 && "Additional args aren't supported yet.");
MachineFunction &MF = MIRBuilder.getMF();
const Function &F = MF.getFunction();
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index fe095414e5172..5438c6f50dea2 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -3657,6 +3657,19 @@ bool SITargetLowering::mayBeEmittedAsTailCall(const CallInst *CI) const {
return true;
}
+namespace {
+// Chain calls have special arguments that we need to handle. These are
+// tagging along at the end of the arguments list(s), after the SGPR and VGPR
+// arguments (index 0 and 1 respectively).
+enum ChainCallArgIdx {
+ Exec = 2,
+ Flags,
+ NumVGPRs,
+ FallbackExec,
+ FallbackCallee
+};
+} // anonymous namespace
+
// The wave scratch offset register is used as the global base pointer.
SDValue SITargetLowering::LowerCall(CallLoweringInfo &CLI,
SmallVectorImpl<SDValue> &InVals) const {
@@ -3665,37 +3678,67 @@ SDValue SITargetLowering::LowerCall(CallLoweringInfo &CLI,
SelectionDAG &DAG = CLI.DAG;
- TargetLowering::ArgListEntry RequestedExec;
- if (IsChainCallConv) {
- // The last argument should be the value that we need to put in EXEC.
- // Pop it out of CLI.Outs and CLI.OutVals before we do any processing so we
- // don't treat it like the rest of the arguments.
- RequestedExec = CLI.Args.back();
- assert(RequestedExec.Node && "No node for EXEC");
+ const SDLoc &DL = CLI.DL;
+ SDValue Chain = CLI.Chain;
+ SDValue Callee = CLI.Callee;
- if (!RequestedExec.Ty->isIntegerTy(Subtarget->getWavefrontSize()))
+ llvm::SmallVector<SDValue, 6> ChainCallSpecialArgs;
+ if (IsChainCallConv) {
+ // The last arguments should be the value that we need to put in EXEC,
+ // followed by the flags and any other arguments with special meanings.
+ // Pop them out of CLI.Outs and CLI.OutVals before we do any processing so
+ // we don't treat them like the "real" arguments.
+ auto RequestedExecIt = std::find_if(
+ CLI.Outs.begin(), CLI.Outs.end(),
+ [](const ISD::OutputArg &Arg) { return Arg.OrigArgIndex == 2; });
+ assert(RequestedExecIt != CLI.Outs.end() && "No node for EXEC");
+
+ size_t SpecialArgsBeginIdx = RequestedExecIt - CLI.Outs.begin();
+ CLI.OutVals.erase(CLI.OutVals.begin() + SpecialArgsBeginIdx,
+ CLI.OutVals.end());
+ CLI.Outs.erase(RequestedExecIt, CLI.Outs.end());
+
+ assert(CLI.Outs.back().OrigArgIndex < 2 &&
+ "Haven't popped all the special args");
+
+ TargetLowering::ArgListEntry RequestedExecArg =
+ CLI.Args[ChainCallArgIdx::Exec];
+ if (!RequestedExecArg.Ty->isIntegerTy(Subtarget->getWavefrontSize()))
return lowerUnhandledCall(CLI, InVals, "Invalid value for EXEC");
- assert(CLI.Outs.back().OrigArgIndex == 2 && "Unexpected last arg");
- CLI.Outs.pop_back();
- CLI.OutVals.pop_back();
+ // Convert constants into TargetConstants, so they become immediate operands
+ // instead of being selected into S_MOV.
+ auto PushNodeOrTargetConstant = [&](TargetLowering::ArgListEntry Arg) {
+ if (auto ArgNode = dyn_cast<ConstantSDNode>(Arg.Node))
+ ChainCallSpecialArgs.push_back(DAG.getTargetConstant(
+ ArgNode->getAPIntValue(), DL, ArgNode->getValueType(0)));
+ else
+ ChainCallSpecialArgs.push_back(Arg.Node);
+ };
- if (RequestedExec.Ty->isIntegerTy(64)) {
- assert(CLI.Outs.back().OrigArgIndex == 2 && "Exec wasn't split up");
- CLI.Outs.pop_back();
- CLI.OutVals.pop_back();
- }
+ PushNodeOrTargetConstant(RequestedExecArg);
+
+ // Process any other special arguments depending on the value of the flags.
+ TargetLowering::ArgListEntry Flags = CLI.Args[ChainCallArgIdx::Flags];
+
+ const APInt &FlagsValue = cast<ConstantSDNode>(Flags.Node)->getAPIntValue();
+ if (FlagsValue.isZero()) {
+ if (CLI.Args.size() > ChainCallArgIdx::Flags + 1)
+ return lowerUnhandledCall(CLI, InVals,
+ "No additional args allowed if flags == 0");
+ } else if (FlagsValue.isOneBitSet(0)) {
+ if (CLI.Args.size() != ChainCallArgIdx::FallbackCallee + 1) {
+ return lowerUnhandledCall(CLI, InVals, "Expected 3 additional args");
+ }
- assert(CLI.Outs.back().OrigArgIndex != 2 &&
- "Haven't popped all the pieces of the EXEC mask");
+ std::for_each(CLI.Args.begin() + ChainCallArgIdx::NumVGPRs,
+ CLI.Args.end(), PushNodeOrTargetConstant);
+ }
}
- const SDLoc &DL = CLI.DL;
SmallVector<ISD::OutputArg, 32> &Outs = CLI.Outs;
SmallVector<SDValue, 32> &OutVals = CLI.OutVals;
SmallVector<ISD::InputArg, 32> &Ins = CLI.Ins;
- SDValue Chain = CLI.Chain;
- SDValue Callee = CLI.Callee;
bool &IsTailCall = CLI.IsTailCall;
bool IsVarArg = CLI.IsVarArg;
bool IsSibCall = false;
@@ -3983,7 +4026,8 @@ SDValue SITargetLowering::LowerCall(CallLoweringInfo &CLI,
}
if (IsChainCallConv)
- Ops.push_back(RequestedExec.Node);
+ Ops.insert(Ops.end(), ChainCallSpecialArgs.begin(),
+ ChainCallSpecialArgs.end());
// Add argument registers to the end of the list so that they are known live
// into the call.
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 63f66023837a2..e5af35f2da686 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -692,36 +692,42 @@ def : GCNPat<
(SI_TCRETURN_GFX Gfx_CCR_SGPR_64:$src0, (i64 0), i32imm:$fpdiff)
>;
-// Pseudo for the llvm.amdgcn.cs.chain intrinsic.
-// This is essentially a tail call, but it also takes a mask to put in EXEC
-// right before jumping to the callee.
-class SI_CS_CHAIN_TC<
+// Pseudos for the llvm.amdgcn.cs.chain intrinsic.
+multiclass SI_CS_CHAIN_TC<
ValueType execvt, Predicate wavesizepred,
- RegisterOperand execrc = getSOPSrcForVT<execvt>.ret>
- : SPseudoInstSI <(outs),
- (ins CCR_SGPR_64:$src0, unknown:$callee, i32imm:$fpdiff, execrc:$exec)> {
- let FixedSize = 0;
- let isCall = 1;
- let isTerminator = 1;
- let isBarrier = 1;
- let isReturn = 1;
- let UseNamedOperandTable = 1;
- let SchedRW = [WriteBranch];
- let isConvergent = 1;
-
- let WaveSizePredicate = wavesizepred;
-}
-
-def SI_CS_CHAIN_TC_W32 : SI_CS_CHAIN_TC<i32, isWave32>;
-def SI_CS_CHAIN_TC_W64 : SI_CS_CHAIN_TC<i64, isWave64>;
+ RegisterOperand execrc = getSOPSrcForVT<execvt>.ret> {
+ let FixedSize = 0,
+ isCall = 1,
+ isTerminator = 1,
+ isBarrier = 1,
+ isReturn = 1,
+ UseNamedOperandTable = 1,
+ SchedRW = [WriteBranch],
+ isConvergent = 1,
+ WaveSizePredicate = wavesizepred in {
+ // This is essentially a tail call, but it also takes a mask to put in EXEC
+ // right before jumping to the callee.
+ def NAME: SPseudoInstSI <(outs),
+ (ins CCR_SGPR_64:$src0, unknown:$callee, i32imm:$fpdiff, execrc:$exec)>;
+
+ // Same as above, but it will first try to reallocate the VGPRs, and choose an
+ // EXEC mask and a callee depending on the success of the reallocation attempt.
+ def _DVGPR : SPseudoInstSI <(outs),
+ (ins CCR_SGPR_64:$src0, i64imm:$callee, i32imm:$fpdiff, execrc:$exec,
+ SSrc_b32:$numvgprs, execrc:$fbexec, CCR_SGPR_64:$fbcallee)>;
+ } // End FixedSize = 0 etc
+}
+
+defm SI_CS_CHAIN_TC_W32 : SI_CS_CHAIN_TC<i32, isWave32>;
+defm SI_CS_CHAIN_TC_W64 : SI_CS_CHAIN_TC<i64, isWave64>;
// Handle selecting direct & indirect calls via SI_CS_CHAIN_TC_W32/64
multiclass si_cs_chain_tc_pattern<
dag callee, ValueType execvt, RegisterOperand execrc, Instruction tc> {
-def : GCNPat<
- (AMDGPUtc_return_chain i64:$src0, callee, (i32 timm:$fpdiff), execvt:$exec),
- (tc CCR_SGPR_64:$src0, callee, i32imm:$fpdiff, execrc:$exec)
->;
+ def : GCNPat<
+ (AMDGPUtc_return_chain i64:$src0, callee, (i32 timm:$fpdiff), execvt:$exec),
+ (tc CCR_SGPR_64:$src0, callee, i32imm:$fpdiff, execrc:$exec)
+ >;
}
multiclass si_cs_chain_tc_patterns<
@@ -736,6 +742,26 @@ multiclass si_cs_chain_tc_patterns<
defm : si_cs_chain_tc_patterns<i32>;
defm : si_cs_chain_tc_patterns<i64>;
+// Match dynamic VGPR case. This is always indirect since we choose the callee
+// dynamically based on the result of the VGPR reall...
[truncated]
|
|
Ping? |
arsenm
reviewed
Mar 10, 2025
Comment on lines
1256
to
1259
| if (!FallbackExecArg.Ty->isIntegerTy(ST.getWavefrontSize())) { | ||
| LLVM_DEBUG(dbgs() << "Bad type for fallback EXEC"); | ||
| return false; | ||
| } |
There was a problem hiding this comment.
For consistency with other contexts, should this accept a wave64 signature when executing wave32, with implicit truncate to i32?
There was a problem hiding this comment.
Ok, but also for consistency we'd probably want to do that for all chain calls, not just when dynamic VGPRs are in use. I'll send a separate patch for that.
llvm/test/CodeGen/AMDGPU/amdgcn-cs-chain-intrinsic-dyn-vgpr-w32.ll
Outdated
Show resolved
Hide resolved
Comment on lines
121
to
124
| if (Op.isReg()) | ||
| MIB.addReg(Op.getReg()); | ||
| else | ||
| MIB->addOperand(Op); |
There was a problem hiding this comment.
Don't understand this, the addOperand always works? MachineInstrBuilder:add is also the same as MIB->addOperand
There was a problem hiding this comment.
This is because the same register may be used twice, and if it has a killed flag then that might get copied to an "earlier" instruction. This is what remove-register-flags.mir is testing. For now I renamed the helper, I should probably add a comment too...
There was a problem hiding this comment.
Comment added. Hope it's clearer now.
| multiclass si_cs_chain_tc_dvgpr_patterns< | ||
| ValueType execvt, RegisterOperand execrc = getSOPSrcForVT<execvt>.ret, | ||
| Instruction tc = SI_CS_CHAIN_TC_W32_DVGPR> { | ||
| let AddedComplexity = 90 in { |
There was a problem hiding this comment.
Why does this need AddedComplexity?
There was a problem hiding this comment.
To be honest, I wrote this >1 year ago, so I don't really remember why I chose to do it this way. I reworked it to use a different ISD node for dynamic VGPRs, I hope it's cleaner this way.
Co-authored-by: Matt Arsenault <[email protected]>
Co-authored-by: Matt Arsenault <[email protected]>
Co-authored-by: Matt Arsenault <[email protected]>
Co-authored-by: Matt Arsenault <[email protected]>
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
arsenm
reviewed
Mar 12, 2025
Co-authored-by: Matt Arsenault <[email protected]>
Co-authored-by: Matt Arsenault <[email protected]>
Co-authored-by: Matt Arsenault <[email protected]>
qiaojbao
added a commit
to GPUOpen-Drivers/llvm-project
that referenced
this pull request
Mar 13, 2025
qiaojbao
added a commit
to GPUOpen-Drivers/llvm-project
that referenced
this pull request
Mar 13, 2025
arsenm
reviewed
Mar 13, 2025
Co-authored-by: Matt Arsenault <[email protected]>
mbrkusanin
reviewed
Mar 14, 2025
arsenm
reviewed
Mar 14, 2025
arsenm
approved these changes
Mar 18, 2025
Comment on lines
+122
to
+125
| if (Op.isReg()) | ||
| MIB.addReg(Op.getReg()); | ||
| else | ||
| MIB.add(Op); |
There was a problem hiding this comment.
There's probably a cleaner way to do this, but we should really finish the removal of kill flags
rovka
force-pushed
the
users/rovka/dvgpr-3
branch
from
3d09c94 to
6b7d174
Compare
Co-authored-by: Matt Arsenault <[email protected]>
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/51/builds/12893 Here is the relevant piece of the build log for the reference |
Thanks @kazutakahirata, you beat me to it! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The llvm.amdgcn.cs.chain intrinsic has a 'flags' operand which may indicate that we want to reallocate the VGPRs before performing the call.
A call with the following arguments:
is supposed to do the following:
This patch implements the dynamic VGPR behaviour by generating an S_ALLOC_VGPR followed by S_CSELECT_B32/64 instructions for the EXEC and callee. The rest of the call sequence is left undisturbed (i.e. identical to the case where the flags are 0 and we don't use dynamic VGPRs). We achieve this by introducing some new pseudos (SI_CS_CHAIN_TC_Wn_DVGPR) which are expanded in the SILateBranchLowering pass, just like the simpler SI_CS_CHAIN_TC_Wn pseudos. The main reason is so that we don't risk other passes (particularly the PostRA scheduler) introducing instructions between the S_ALLOC_VGPR and the jump. Such instructions might end up using VGPRs that have been deallocated, or the wrong EXEC mask. Once the whole backend treats S_ALLOC_VGPR and changes to EXEC as barriers for instructions that use VGPRs, we could in principle move the expansion earlier (but in the absence of a good reason for that my personal preference is to keep it later in order to make debugging easier).
Since the expansion happens after register allocation, we're careful to select constants to immediate operands instead of letting ISel generate S_MOVs which could interfere with register allocation (i.e. make it look like we need more registers than we actually do).
For GFX12, S_ALLOC_VGPR only works in wave32 mode, so we bail out during ISel in wave64 mode. However, we can define the pseudos for wave64 too so it's easy to handle if future generations support it.