Skip to content

[AMDGPU] GFX12: select @llvm.prefetch intrinsic #74576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 15, 2023
Merged

[AMDGPU] GFX12: select @llvm.prefetch intrinsic #74576

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2375974
[AMDGPU] GFX12: select @llvm.prefetch intrinsic
rampitec Jun 28, 2022
818c33d
Merge remote-tracking branch 'upstream-repo/main' into masikora/upstr…
mariusz-sikora-at-amd Dec 7, 2023
3d56730
Stop generating _PC_REL form
mariusz-sikora-at-amd Dec 7, 2023
a4366be
Remove PrefetchGetPcPat
mariusz-sikora-at-amd Dec 15, 2023
c2b2ede
Merge remote-tracking branch main into upstream-gfx12-sh-prefetch-ops…
mariusz-sikora-at-amd Dec 15, 2023
0598ae2
G_PREFETCH part
mariusz-sikora-at-amd Dec 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3101,6 +3101,24 @@ void AMDGPURegisterBankInfo::applyMappingImpl(
applyDefaultMapping(OpdMapper);
constrainOpWithReadfirstlane(B, MI, 8); // M0
return;
case Intrinsic::prefetch: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have a G_ instruction.

Why erase here and not in the legalizer?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should have G_PREFETCH, but we do not currently have it.
Then I honestly do not remember why do I erase it here. Likely because I am erasing it here for VGPRBank anyway.

if (!Subtarget.hasPrefetch()) {
MI.eraseFromParent();
return;
}
unsigned PtrBank =
getRegBankID(MI.getOperand(1).getReg(), MRI, AMDGPU::SGPRRegBankID);
if (PtrBank == AMDGPU::VGPRRegBankID) {
MI.eraseFromParent();
return;
}
// FIXME: There is currently no support for prefetch in global isel.
// There is no node equivalence and what's worse there is no MMO produced
// for a prefetch on global isel path.
// Prefetch does not affect execution so erase it for now.
MI.eraseFromParent();
return;
}
default: {
if (const AMDGPU::RsrcIntrinsic *RSrcIntrin =
AMDGPU::lookupRsrcIntrinsic(IntrID)) {
Expand Down Expand Up @@ -4830,6 +4848,9 @@ AMDGPURegisterBankInfo::getInstrMapping(const MachineInstr &MI) const {
getVGPROpMapping(MI.getOperand(5).getReg(), MRI, *TRI); // %data1
break;
}
case Intrinsic::prefetch:
OpdsMapping[1] = getSGPROpMapping(MI.getOperand(1).getReg(), MRI, *TRI);
break;

default:
return getInvalidInstructionMapping();
Expand Down
2 changes: 2 additions & 0 deletions llvm/lib/Target/AMDGPU/GCNSubtarget.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -828,6 +828,8 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,

bool hasInstPrefetch() const { return getGeneration() >= GFX10; }

bool hasPrefetch() const { return GFX12Insts; }

// Scratch is allocated in 256 dword per wave blocks for the entire
// wavefront. When viewed from the perspective of an arbitrary workitem, this
// is 4-byte aligned.
Expand Down
22 changes: 22 additions & 0 deletions llvm/lib/Target/AMDGPU/SIISelLowering.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -763,6 +763,9 @@ SITargetLowering::SITargetLowering(const TargetMachine &TM,
if (Subtarget->hasMad64_32())
setOperationAction({ISD::SMUL_LOHI, ISD::UMUL_LOHI}, MVT::i32, Custom);

if (Subtarget->hasPrefetch())
setOperationAction(ISD::PREFETCH, MVT::Other, Custom);

setOperationAction(ISD::INTRINSIC_WO_CHAIN,
{MVT::Other, MVT::f32, MVT::v4f32, MVT::i16, MVT::f16,
MVT::v2i16, MVT::v2f16, MVT::i128},
Expand Down Expand Up @@ -3858,6 +3861,23 @@ SDValue SITargetLowering::lowerGET_ROUNDING(SDValue Op,
return DAG.getMergeValues({Result, GetReg.getValue(1)}, SL);
}

SDValue SITargetLowering::lowerPREFETCH(SDValue Op, SelectionDAG &DAG) const {
if (Op->isDivergent())
return SDValue();

switch (cast<MemSDNode>(Op)->getAddressSpace()) {
case AMDGPUAS::FLAT_ADDRESS:
case AMDGPUAS::GLOBAL_ADDRESS:
case AMDGPUAS::CONSTANT_ADDRESS:
case AMDGPUAS::CONSTANT_ADDRESS_32BIT:
break;
default:
return SDValue();
}

return Op;
}

Register SITargetLowering::getRegisterByName(const char* RegName, LLT VT,
const MachineFunction &MF) const {
Register Reg = StringSwitch<Register>(RegName)
Expand Down Expand Up @@ -5395,6 +5415,8 @@ SDValue SITargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
return LowerSTACKSAVE(Op, DAG);
case ISD::GET_ROUNDING:
return lowerGET_ROUNDING(Op, DAG);
case ISD::PREFETCH:
return lowerPREFETCH(Op, DAG);
}
return SDValue();
}
Expand Down
2 changes: 2 additions & 0 deletions llvm/lib/Target/AMDGPU/SIISelLowering.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,8 @@ class SITargetLowering final : public AMDGPUTargetLowering {
SDValue LowerSTACKSAVE(SDValue Op, SelectionDAG &DAG) const;
SDValue lowerGET_ROUNDING(SDValue Op, SelectionDAG &DAG) const;

SDValue lowerPREFETCH(SDValue Op, SelectionDAG &DAG) const;

Register getRegisterByName(const char* RegName, LLT VT,
const MachineFunction &MF) const override;

Expand Down
2 changes: 2 additions & 0 deletions llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,8 @@ bool SIInstrInfo::getMemOperandsWithOffsetWidth(
Offset = OffsetOp ? OffsetOp->getImm() : 0;
// Get appropriate operand, and compute width accordingly.
DataOpIdx = AMDGPU::getNamedOperandIdx(Opc, AMDGPU::OpName::sdst);
if (DataOpIdx == -1)
return false;
Width = getOpSize(LdSt, DataOpIdx);
return true;
}
Expand Down
12 changes: 12 additions & 0 deletions llvm/lib/Target/AMDGPU/SIInstructions.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3164,6 +3164,18 @@ def : GCNPat <
(as_i1timm $bound_ctrl))
>;

class SMPrefetchGetPcPat<string type, int cache_type> : GCNPat <
(prefetch (i64 imm:$offset), timm, timm, (i32 cache_type)),
(!cast<SM_Prefetch_Pseudo>("S_PREFETCH_"#type) (S_ADD_U64_PSEUDO (S_GETPC_B64), $offset),
(i32 20), (i32 SGPR_NULL), (i8 0))
// Offset 20 should roughly adjust getpc sequence length.
> {
let AddedComplexity = 9;
}

def : SMPrefetchGetPcPat<"INST", 0>;
def : SMPrefetchGetPcPat<"DATA", 1>;

//===----------------------------------------------------------------------===//
// Fract Patterns
//===----------------------------------------------------------------------===//
Expand Down
34 changes: 34 additions & 0 deletions llvm/lib/Target/AMDGPU/SMInstructions.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -814,6 +814,14 @@ def smrd_load : PatFrag <(ops node:$ptr), (load node:$ptr), [{ return isUniformL
}];
}

def smrd_prefetch : PatFrag <(ops node:$ptr, node:$rw, node:$loc, node:$type),
(prefetch node:$ptr, node:$rw, node:$loc, node:$type),
[{ return !N->getOperand(1)->isDivergent();}]> {
let GISelPredicateCode = [{
return isInstrUniform(MI);
}];
}

def SMRDImm : ComplexPattern<iPTR, 2, "SelectSMRDImm">;
def SMRDImm32 : ComplexPattern<iPTR, 2, "SelectSMRDImm32">;
def SMRDSgpr : ComplexPattern<iPTR, 2, "SelectSMRDSgpr">;
Expand Down Expand Up @@ -959,6 +967,32 @@ def : GCNPat <
}
} // let OtherPredicates = [HasShaderCyclesRegister]

def SIMM24bitPtr : ImmLeaf <iPTR,
[{return isInt<24>(Imm);}]
>;

multiclass SMPrefetchPat<string type, int cache_type> {
def : GCNPat <
(smrd_prefetch (SMRDImm i64:$sbase, i32:$offset), timm, timm, (i32 cache_type)),
(!cast<SM_Prefetch_Pseudo>("S_PREFETCH_"#type) $sbase, $offset, (i32 SGPR_NULL), (i8 0))
>;

def : GCNPat <
(smrd_prefetch (i64 SReg_64:$sbase), timm, timm, (i32 cache_type)),
(!cast<SM_Prefetch_Pseudo>("S_PREFETCH_"#type) $sbase, 0, (i32 SGPR_NULL), (i8 0))
>;

def : GCNPat <
(prefetch SIMM24bitPtr:$offset, timm, timm, (i32 cache_type)),
(!cast<SM_Prefetch_Pseudo>("S_PREFETCH_"#type#"_PC_REL") (as_i32timm $offset), (i32 SGPR_NULL), (i8 0))
> {
let AddedComplexity = 10;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this pattern. (prefetch 0x12345678, 0, 0, 0) should prefetch from absolute address 0x12345678, but this pattern selects S_PREFETCH_*_PC_REL 0x12345678, null, 0 which will prefetch from pc+0x12345678, which is different. @rampitec can you explain?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you do not have pointer (essentially provide a null to the prefetch intrinsic as a base pointer) this pc_rel pattern will be used. It may have no value as a data prefetch, but makes sense as inst prefetch. Say if you want to prefetch a next page of code. Like this:

define amdgpu_ps void @prefetch_inst_pc_rel_offset() {
; GFX12-SDAG-LABEL: prefetch_inst_pc_rel_offset:
; GFX12-SDAG:       ; %bb.0: ; %entry
; GFX12-SDAG-NEXT:    s_prefetch_inst_pc_rel 0x80, null, 0
; GFX12-SDAG-NEXT:    s_endpgm
entry:
  %gep = getelementptr i8, ptr addrspace(4) null, i32 128
  tail call void @llvm.prefetch.p4(ptr addrspace(4) %gep, i32 0, i32 0, i32 0)
  ret void
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would interpret this as using the absolute address, you would need something else to represent a PC relative input

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe for now I will remove PC_REL part.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefetch on an absolute address is practically useless.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that is how llvm.prefetch is defined: "address is the address to be prefetched". A different operation should use a different intrinsic.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you want a target intrinsic?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really don't know. What would the use cases look like? Maybe it could be a generic intrinsic, if there is consensus that it is useful.

For the existing llvm.prefetch intrinsic, the only useful case I think of for instruction prefetching is:

define @f0() {
  call @llvm.prefetch(@f1, ...) 
  ...
  call @f1()
}
define @f1() { ... }

to prefetch the code at the start of a function you are going to call. We could codegen that case using the _pc_rel form of the instruction.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think we need to use PC_REL form to prefetch on a function's address. The instruction can take full 64-bit address, so one can just use this address. My understanding that PC_REL form can be useful if you expect something like a huge loop or a local branch and want to prefetch something like 1K from the PC. I am not sure though how useful this can be at a high language level or even in IR.

}

defm : SMPrefetchPat<"INST", 0>;
defm : SMPrefetchPat<"DATA", 1>;

//===----------------------------------------------------------------------===//
// GFX10.
//===----------------------------------------------------------------------===//
Expand Down
Loading