[ARM] Fix calling convention for __fp16 with big-endian #126741
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AAPCS32 defines the fp16 and bf16 types as being passed as if they were extended to 32 bits, with the high 16 bits being unspecified. The extension is specified as happening as-if it was done in a register, which means that for big endian targets, the actual value gets passed in the higher adressed half of the stack slot, instead of the lower addressed half as for little endian. Previously, we were passing these types as a 16 bit stack slot, which worked for little endian because every later stack slot would be 4-byte aligned leaving the 2 byte gap, but was incorect for big endian. There was also a selection failure when passing bf16 values on the stack without the bf16 hardware extension, this patch fixes that as a side effect by initially emitting the load as i32, and optimising to a bf16 load if the type is legal.
|
@llvm/pr-subscribers-backend-arm Author: Oliver Stannard (ostannard) ChangesAAPCS32 defines the fp16 and bf16 types as being passed as if they were extended to 32 bits, with the high 16 bits being unspecified. The extension is specified as happening as-if it was done in a register, which means that for big endian targets, the actual value gets passed in the higher addressed half of the stack slot, instead of the lower addressed half as for little endian. Previously, for targets with the fp16 extension, we were passing these types as a 16 bit stack slot, which worked for little endian because every later stack slot would be 4-byte aligned leaving the 2 byte gap, but was incorrect for big endian. There was also a selection failure when passing bf16 values on the stack without the bf16 hardware extension, this patch fixes that as a side effect by initially emitting the load as i32, and optimising to a bf16 load if the type is legal. Patch is 25.67 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/126741.diff 5 Files Affected:
diff --git a/llvm/lib/Target/ARM/ARMCallingConv.cpp b/llvm/lib/Target/ARM/ARMCallingConv.cpp
index 66a76a8c7a95a1..a206e993394f7d 100644
--- a/llvm/lib/Target/ARM/ARMCallingConv.cpp
+++ b/llvm/lib/Target/ARM/ARMCallingConv.cpp
@@ -298,7 +298,8 @@ static bool CustomAssignInRegList(unsigned ValNo, MVT ValVT, MVT LocVT,
static bool CC_ARM_AAPCS_Custom_f16(unsigned ValNo, MVT ValVT, MVT LocVT,
CCValAssign::LocInfo LocInfo,
ISD::ArgFlagsTy ArgFlags, CCState &State) {
- // f16 arguments are extended to i32 and assigned to a register in [r0, r3]
+ // f16 and bf16 arguments are extended to i32 and assigned to a register in
+ // [r0, r3].
return CustomAssignInRegList(ValNo, ValVT, MVT::i32, LocInfo, State,
RRegList);
}
@@ -307,10 +308,25 @@ static bool CC_ARM_AAPCS_VFP_Custom_f16(unsigned ValNo, MVT ValVT, MVT LocVT,
CCValAssign::LocInfo LocInfo,
ISD::ArgFlagsTy ArgFlags,
CCState &State) {
- // f16 arguments are extended to f32 and assigned to a register in [s0, s15]
+ // f16 and bf16 arguments are extended to f32 and assigned to a register in
+ // [s0, s15].
return CustomAssignInRegList(ValNo, ValVT, MVT::f32, LocInfo, State,
SRegList);
}
+static bool CC_ARM_AAPCS_Common_Custom_f16_Stack(unsigned ValNo, MVT ValVT,
+ MVT LocVT,
+ CCValAssign::LocInfo LocInfo,
+ ISD::ArgFlagsTy ArgFlags,
+ CCState &State) {
+ // f16 and bf16 (if not passed in a register) are assigned to a 32-bit stack
+ // slot, with the most-significant 16 bits unspecified. The 32-bit slot is
+ // important to make sure that the byte ordering is correct for big endian
+ // targets.
+ State.addLoc(CCValAssign::getCustomMem(
+ ValNo, ValVT, State.AllocateStack(4, Align(4)), MVT::i32, LocInfo));
+ return true;
+}
+
// Include the table generated calling convention implementations.
#include "ARMGenCallingConv.inc"
diff --git a/llvm/lib/Target/ARM/ARMCallingConv.td b/llvm/lib/Target/ARM/ARMCallingConv.td
index 27f175a7003366..f1ab1c31037401 100644
--- a/llvm/lib/Target/ARM/ARMCallingConv.td
+++ b/llvm/lib/Target/ARM/ARMCallingConv.td
@@ -139,7 +139,8 @@ def CC_ARM_AAPCS_Common : CallingConv<[
CCIfType<[i32], CCIfAlign<"8", CCAssignToStackWithShadow<4, 8, [R0, R1, R2, R3]>>>,
CCIfType<[i32], CCAssignToStackWithShadow<4, 4, [R0, R1, R2, R3]>>,
- CCIfType<[f16, bf16, f32], CCAssignToStackWithShadow<4, 4, [Q0, Q1, Q2, Q3]>>,
+ CCIfType<[f32], CCAssignToStackWithShadow<4, 4, [Q0, Q1, Q2, Q3]>>,
+ CCIfType<[f16, bf16], CCCustom<"CC_ARM_AAPCS_Common_Custom_f16_Stack">>,
CCIfType<[f64], CCAssignToStackWithShadow<8, 8, [Q0, Q1, Q2, Q3]>>,
CCIfType<[v2f64], CCIfAlign<"16",
CCAssignToStackWithShadow<16, 16, [Q0, Q1, Q2, Q3]>>>,
diff --git a/llvm/lib/Target/ARM/ARMISelLowering.cpp b/llvm/lib/Target/ARM/ARMISelLowering.cpp
index bd8d6079e1ba88..a856de49622172 100644
--- a/llvm/lib/Target/ARM/ARMISelLowering.cpp
+++ b/llvm/lib/Target/ARM/ARMISelLowering.cpp
@@ -4759,6 +4759,24 @@ SDValue ARMTargetLowering::LowerFormalArguments(
VA.getLocMemOffset(), Flags.getByValSize());
InVals.push_back(DAG.getFrameIndex(FrameIndex, PtrVT));
CCInfo.nextInRegsParam();
+ } else if (VA.needsCustom() && (VA.getValVT() == MVT::f16 ||
+ VA.getValVT() == MVT::bf16)) {
+ // f16 and bf16 values are passed in the least-significant half of
+ // a 4 byte stack slot. This is done as-if the extension was done
+ // in a 32-bit register, so the actual bytes used for the value
+ // differ between little and big endian.
+ unsigned FIOffset = VA.getLocMemOffset();
+ int FI = MFI.CreateFixedObject(VA.getLocVT().getSizeInBits()/8,
+ FIOffset, true);
+
+ // Create load nodes to retrieve arguments from the stack.
+ SDValue FIN = DAG.getFrameIndex(FI, PtrVT);
+ SDValue Load = DAG.getLoad(VA.getLocVT(), dl, Chain, FIN,
+ MachinePointerInfo::getFixedStack(
+ DAG.getMachineFunction(), FI));
+ InVals.push_back(
+ MoveToHPR(dl, DAG, VA.getLocVT(), VA.getValVT(), Load));
+
} else {
unsigned FIOffset = VA.getLocMemOffset();
int FI = MFI.CreateFixedObject(VA.getLocVT().getSizeInBits()/8,
@@ -15257,10 +15275,19 @@ static SDValue PerformVMOVhrCombine(SDNode *N,
// fold (VMOVhr (load x)) -> (load (f16*)x)
if (LoadSDNode *LN0 = dyn_cast<LoadSDNode>(Op0)) {
if (LN0->hasOneUse() && LN0->isUnindexed() &&
- LN0->getMemoryVT() == MVT::i16) {
- SDValue Load =
- DCI.DAG.getLoad(N->getValueType(0), SDLoc(N), LN0->getChain(),
- LN0->getBasePtr(), LN0->getMemOperand());
+ (LN0->getMemoryVT() == MVT::i16 || LN0->getMemoryVT() == MVT::i32)) {
+ SDValue Addr = LN0->getBasePtr();
+ unsigned PtrOffset = 0;
+ if (DCI.DAG.getDataLayout().isBigEndian() &&
+ LN0->getMemoryVT() == MVT::i32) {
+ PtrOffset = 2;
+ Addr = DCI.DAG.getObjectPtrOffset(SDLoc(N), Addr,
+ TypeSize::getFixed(PtrOffset));
+ }
+ SDValue Load = DCI.DAG.getLoad(
+ N->getValueType(0), SDLoc(N), LN0->getChain(), Addr,
+ LN0->getPointerInfo().getWithOffset(PtrOffset), LN0->getAlign(),
+ LN0->getMemOperand()->getFlags(), LN0->getAAInfo());
DCI.DAG.ReplaceAllUsesOfValueWith(SDValue(N, 0), Load.getValue(0));
DCI.DAG.ReplaceAllUsesOfValueWith(Op0.getValue(1), Load.getValue(1));
return Load;
diff --git a/llvm/test/CodeGen/Thumb2/bf16-pcs.ll b/llvm/test/CodeGen/Thumb2/bf16-pcs.ll
new file mode 100644
index 00000000000000..ccf9400f613208
--- /dev/null
+++ b/llvm/test/CodeGen/Thumb2/bf16-pcs.ll
@@ -0,0 +1,316 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=armv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16 | FileCheck %s --check-prefix=LE
+; RUN: llc -mtriple=armebv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16 | FileCheck %s --check-prefix=BE
+; RUN: llc -mtriple=armv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16,+bf16 | FileCheck %s --check-prefix=LE-BF16
+; RUN: llc -mtriple=armebv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16,+bf16 | FileCheck %s --check-prefix=BE-BF16
+
+define arm_aapcscc bfloat @callee_soft_bfloat_in_reg(bfloat %f) {
+; LE-LABEL: callee_soft_bfloat_in_reg:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: bx lr
+;
+; BE-LABEL: callee_soft_bfloat_in_reg:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: bx lr
+;
+; LE-BF16-LABEL: callee_soft_bfloat_in_reg:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: .pad #4
+; LE-BF16-NEXT: sub sp, #4
+; LE-BF16-NEXT: strh.w r0, [sp, #2]
+; LE-BF16-NEXT: ldrh.w r0, [sp, #2]
+; LE-BF16-NEXT: add sp, #4
+; LE-BF16-NEXT: bx lr
+;
+; BE-BF16-LABEL: callee_soft_bfloat_in_reg:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: .pad #4
+; BE-BF16-NEXT: sub sp, #4
+; BE-BF16-NEXT: strh.w r0, [sp, #2]
+; BE-BF16-NEXT: ldrh.w r0, [sp, #2]
+; BE-BF16-NEXT: add sp, #4
+; BE-BF16-NEXT: bx lr
+entry:
+ ret bfloat %f
+}
+
+define void @caller_soft_bfloat_in_reg() {
+; LE-LABEL: caller_soft_bfloat_in_reg:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: .save {r7, lr}
+; LE-NEXT: push {r7, lr}
+; LE-NEXT: mov.w r0, #16256
+; LE-NEXT: bl callee_soft_bfloat_in_reg
+; LE-NEXT: pop {r7, pc}
+;
+; BE-LABEL: caller_soft_bfloat_in_reg:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: .save {r7, lr}
+; BE-NEXT: push {r7, lr}
+; BE-NEXT: mov.w r0, #16256
+; BE-NEXT: bl callee_soft_bfloat_in_reg
+; BE-NEXT: pop {r7, pc}
+;
+; LE-BF16-LABEL: caller_soft_bfloat_in_reg:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: .save {r7, lr}
+; LE-BF16-NEXT: push {r7, lr}
+; LE-BF16-NEXT: mov.w r0, #16256
+; LE-BF16-NEXT: bl callee_soft_bfloat_in_reg
+; LE-BF16-NEXT: pop {r7, pc}
+;
+; BE-BF16-LABEL: caller_soft_bfloat_in_reg:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: .save {r7, lr}
+; BE-BF16-NEXT: push {r7, lr}
+; BE-BF16-NEXT: mov.w r0, #16256
+; BE-BF16-NEXT: bl callee_soft_bfloat_in_reg
+; BE-BF16-NEXT: pop {r7, pc}
+entry:
+ %ret = call arm_aapcscc bfloat @callee_soft_bfloat_in_reg(bfloat 1.0)
+ ret void
+}
+
+define arm_aapcscc bfloat @callee_soft_bfloat_on_stack(float %r0, float %r1, float %r2, float %r3, bfloat %f) {
+; LE-LABEL: callee_soft_bfloat_on_stack:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: ldr r0, [sp]
+; LE-NEXT: bx lr
+;
+; BE-LABEL: callee_soft_bfloat_on_stack:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: ldr r0, [sp]
+; BE-NEXT: bx lr
+;
+; LE-BF16-LABEL: callee_soft_bfloat_on_stack:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: ldrh.w r0, [sp]
+; LE-BF16-NEXT: bx lr
+;
+; BE-BF16-LABEL: callee_soft_bfloat_on_stack:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: ldrh.w r0, [sp, #2]
+; BE-BF16-NEXT: bx lr
+entry:
+ ret bfloat %f
+}
+
+define void @caller_soft_bfloat_on_stack() {
+; LE-LABEL: caller_soft_bfloat_on_stack:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: .save {r7, lr}
+; LE-NEXT: push {r7, lr}
+; LE-NEXT: .pad #8
+; LE-NEXT: sub sp, #8
+; LE-NEXT: mov.w r0, #16256
+; LE-NEXT: str r0, [sp]
+; LE-NEXT: bl callee_soft_bfloat_on_stack
+; LE-NEXT: add sp, #8
+; LE-NEXT: pop {r7, pc}
+;
+; BE-LABEL: caller_soft_bfloat_on_stack:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: .save {r7, lr}
+; BE-NEXT: push {r7, lr}
+; BE-NEXT: .pad #8
+; BE-NEXT: sub sp, #8
+; BE-NEXT: mov.w r0, #16256
+; BE-NEXT: str r0, [sp]
+; BE-NEXT: bl callee_soft_bfloat_on_stack
+; BE-NEXT: add sp, #8
+; BE-NEXT: pop {r7, pc}
+;
+; LE-BF16-LABEL: caller_soft_bfloat_on_stack:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: .save {r7, lr}
+; LE-BF16-NEXT: push {r7, lr}
+; LE-BF16-NEXT: .pad #8
+; LE-BF16-NEXT: sub sp, #8
+; LE-BF16-NEXT: mov.w r0, #16256
+; LE-BF16-NEXT: str r0, [sp]
+; LE-BF16-NEXT: bl callee_soft_bfloat_on_stack
+; LE-BF16-NEXT: add sp, #8
+; LE-BF16-NEXT: pop {r7, pc}
+;
+; BE-BF16-LABEL: caller_soft_bfloat_on_stack:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: .save {r7, lr}
+; BE-BF16-NEXT: push {r7, lr}
+; BE-BF16-NEXT: .pad #8
+; BE-BF16-NEXT: sub sp, #8
+; BE-BF16-NEXT: mov.w r0, #16256
+; BE-BF16-NEXT: str r0, [sp]
+; BE-BF16-NEXT: bl callee_soft_bfloat_on_stack
+; BE-BF16-NEXT: add sp, #8
+; BE-BF16-NEXT: pop {r7, pc}
+entry:
+ %ret = call arm_aapcscc bfloat @callee_soft_bfloat_on_stack(float undef, float undef, float undef, float undef, bfloat 1.0)
+ ret void
+}
+
+define arm_aapcs_vfpcc bfloat @callee_hard_bfloat_in_reg(bfloat %f) {
+; LE-LABEL: callee_hard_bfloat_in_reg:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: bx lr
+;
+; BE-LABEL: callee_hard_bfloat_in_reg:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: bx lr
+;
+; LE-BF16-LABEL: callee_hard_bfloat_in_reg:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: .pad #4
+; LE-BF16-NEXT: sub sp, #4
+; LE-BF16-NEXT: vmov r0, s0
+; LE-BF16-NEXT: strh.w r0, [sp, #2]
+; LE-BF16-NEXT: ldrh.w r0, [sp, #2]
+; LE-BF16-NEXT: vmov s0, r0
+; LE-BF16-NEXT: add sp, #4
+; LE-BF16-NEXT: bx lr
+;
+; BE-BF16-LABEL: callee_hard_bfloat_in_reg:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: .pad #4
+; BE-BF16-NEXT: sub sp, #4
+; BE-BF16-NEXT: vmov r0, s0
+; BE-BF16-NEXT: strh.w r0, [sp, #2]
+; BE-BF16-NEXT: ldrh.w r0, [sp, #2]
+; BE-BF16-NEXT: vmov s0, r0
+; BE-BF16-NEXT: add sp, #4
+; BE-BF16-NEXT: bx lr
+entry:
+ ret bfloat %f
+}
+
+define void @caller_hard_bfloat_in_reg() {
+; LE-LABEL: caller_hard_bfloat_in_reg:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: .save {r7, lr}
+; LE-NEXT: push {r7, lr}
+; LE-NEXT: vldr s0, .LCPI5_0
+; LE-NEXT: bl callee_hard_bfloat_in_reg
+; LE-NEXT: pop {r7, pc}
+; LE-NEXT: .p2align 2
+; LE-NEXT: @ %bb.1:
+; LE-NEXT: .LCPI5_0:
+; LE-NEXT: .long 0x00003f80 @ float 2.27795078E-41
+;
+; BE-LABEL: caller_hard_bfloat_in_reg:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: .save {r7, lr}
+; BE-NEXT: push {r7, lr}
+; BE-NEXT: vldr s0, .LCPI5_0
+; BE-NEXT: bl callee_hard_bfloat_in_reg
+; BE-NEXT: pop {r7, pc}
+; BE-NEXT: .p2align 2
+; BE-NEXT: @ %bb.1:
+; BE-NEXT: .LCPI5_0:
+; BE-NEXT: .long 0x00003f80 @ float 2.27795078E-41
+;
+; LE-BF16-LABEL: caller_hard_bfloat_in_reg:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: .save {r7, lr}
+; LE-BF16-NEXT: push {r7, lr}
+; LE-BF16-NEXT: vldr s0, .LCPI5_0
+; LE-BF16-NEXT: bl callee_hard_bfloat_in_reg
+; LE-BF16-NEXT: pop {r7, pc}
+; LE-BF16-NEXT: .p2align 2
+; LE-BF16-NEXT: @ %bb.1:
+; LE-BF16-NEXT: .LCPI5_0:
+; LE-BF16-NEXT: .long 0x00003f80 @ float 2.27795078E-41
+;
+; BE-BF16-LABEL: caller_hard_bfloat_in_reg:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: .save {r7, lr}
+; BE-BF16-NEXT: push {r7, lr}
+; BE-BF16-NEXT: vldr s0, .LCPI5_0
+; BE-BF16-NEXT: bl callee_hard_bfloat_in_reg
+; BE-BF16-NEXT: pop {r7, pc}
+; BE-BF16-NEXT: .p2align 2
+; BE-BF16-NEXT: @ %bb.1:
+; BE-BF16-NEXT: .LCPI5_0:
+; BE-BF16-NEXT: .long 0x00003f80 @ float 2.27795078E-41
+entry:
+ %ret = call arm_aapcs_vfpcc bfloat @callee_hard_bfloat_in_reg(bfloat 1.0)
+ ret void
+}
+
+define arm_aapcs_vfpcc bfloat @callee_hard_bfloat_on_stack(float %s0, float %s1, float %s2, float %s3, float %s4, float %s5, float %s6, float %s7,float %s8, float %s9, float %s10, float %s11, float %s12, float %s13, float %s14, float %s15, bfloat %f) {
+; LE-LABEL: callee_hard_bfloat_on_stack:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: vldr s0, [sp]
+; LE-NEXT: bx lr
+;
+; BE-LABEL: callee_hard_bfloat_on_stack:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: vldr s0, [sp]
+; BE-NEXT: bx lr
+;
+; LE-BF16-LABEL: callee_hard_bfloat_on_stack:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: ldrh.w r0, [sp]
+; LE-BF16-NEXT: vmov s0, r0
+; LE-BF16-NEXT: bx lr
+;
+; BE-BF16-LABEL: callee_hard_bfloat_on_stack:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: ldrh.w r0, [sp, #2]
+; BE-BF16-NEXT: vmov s0, r0
+; BE-BF16-NEXT: bx lr
+entry:
+ ret bfloat %f
+}
+
+
+define void @caller_hard_bfloat_on_stack() {
+; LE-LABEL: caller_hard_bfloat_on_stack:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: .save {r7, lr}
+; LE-NEXT: push {r7, lr}
+; LE-NEXT: .pad #8
+; LE-NEXT: sub sp, #8
+; LE-NEXT: mov.w r0, #16256
+; LE-NEXT: str r0, [sp]
+; LE-NEXT: bl callee_hard_bfloat_on_stack
+; LE-NEXT: add sp, #8
+; LE-NEXT: pop {r7, pc}
+;
+; BE-LABEL: caller_hard_bfloat_on_stack:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: .save {r7, lr}
+; BE-NEXT: push {r7, lr}
+; BE-NEXT: .pad #8
+; BE-NEXT: sub sp, #8
+; BE-NEXT: mov.w r0, #16256
+; BE-NEXT: str r0, [sp]
+; BE-NEXT: bl callee_hard_bfloat_on_stack
+; BE-NEXT: add sp, #8
+; BE-NEXT: pop {r7, pc}
+;
+; LE-BF16-LABEL: caller_hard_bfloat_on_stack:
+; LE-BF16: @ %bb.0: @ %entry
+; LE-BF16-NEXT: .save {r7, lr}
+; LE-BF16-NEXT: push {r7, lr}
+; LE-BF16-NEXT: .pad #8
+; LE-BF16-NEXT: sub sp, #8
+; LE-BF16-NEXT: mov.w r0, #16256
+; LE-BF16-NEXT: str r0, [sp]
+; LE-BF16-NEXT: bl callee_hard_bfloat_on_stack
+; LE-BF16-NEXT: add sp, #8
+; LE-BF16-NEXT: pop {r7, pc}
+;
+; BE-BF16-LABEL: caller_hard_bfloat_on_stack:
+; BE-BF16: @ %bb.0: @ %entry
+; BE-BF16-NEXT: .save {r7, lr}
+; BE-BF16-NEXT: push {r7, lr}
+; BE-BF16-NEXT: .pad #8
+; BE-BF16-NEXT: sub sp, #8
+; BE-BF16-NEXT: mov.w r0, #16256
+; BE-BF16-NEXT: str r0, [sp]
+; BE-BF16-NEXT: bl callee_hard_bfloat_on_stack
+; BE-BF16-NEXT: add sp, #8
+; BE-BF16-NEXT: pop {r7, pc}
+entry:
+ %ret = call arm_aapcs_vfpcc bfloat @callee_hard_bfloat_on_stack(float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, float undef, bfloat 1.0)
+ ret void
+}
diff --git a/llvm/test/CodeGen/Thumb2/fp16-pcs.ll b/llvm/test/CodeGen/Thumb2/fp16-pcs.ll
new file mode 100644
index 00000000000000..7785eb8904b78e
--- /dev/null
+++ b/llvm/test/CodeGen/Thumb2/fp16-pcs.ll
@@ -0,0 +1,288 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=armv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16 | FileCheck %s --check-prefix=LE
+; RUN: llc -mtriple=armebv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16 | FileCheck %s --check-prefix=BE
+; RUN: llc -mtriple=armv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16,+fullfp16 | FileCheck %s --check-prefix=LE-FP16
+; RUN: llc -mtriple=armebv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16,+fullfp16 | FileCheck %s --check-prefix=BE-FP16
+
+define arm_aapcscc half @callee_soft_half_in_reg(half %f) {
+; LE-LABEL: callee_soft_half_in_reg:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: bx lr
+;
+; BE-LABEL: callee_soft_half_in_reg:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: bx lr
+;
+; LE-FP16-LABEL: callee_soft_half_in_reg:
+; LE-FP16: @ %bb.0: @ %entry
+; LE-FP16-NEXT: vmov.f16 s0, r0
+; LE-FP16-NEXT: vmov r0, s0
+; LE-FP16-NEXT: bx lr
+;
+; BE-FP16-LABEL: callee_soft_half_in_reg:
+; BE-FP16: @ %bb.0: @ %entry
+; BE-FP16-NEXT: vmov.f16 s0, r0
+; BE-FP16-NEXT: vmov r0, s0
+; BE-FP16-NEXT: bx lr
+entry:
+ ret half %f
+}
+
+define void @caller_soft_half_in_reg() {
+; LE-LABEL: caller_soft_half_in_reg:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: .save {r7, lr}
+; LE-NEXT: push {r7, lr}
+; LE-NEXT: mov.w r0, #15360
+; LE-NEXT: bl callee_soft_half_in_reg
+; LE-NEXT: pop {r7, pc}
+;
+; BE-LABEL: caller_soft_half_in_reg:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: .save {r7, lr}
+; BE-NEXT: push {r7, lr}
+; BE-NEXT: mov.w r0, #15360
+; BE-NEXT: bl callee_soft_half_in_reg
+; BE-NEXT: pop {r7, pc}
+;
+; LE-FP16-LABEL: caller_soft_half_in_reg:
+; LE-FP16: @ %bb.0: @ %entry
+; LE-FP16-NEXT: .save {r7, lr}
+; LE-FP16-NEXT: push {r7, lr}
+; LE-FP16-NEXT: mov.w r0, #15360
+; LE-FP16-NEXT: bl callee_soft_half_in_reg
+; LE-FP16-NEXT: pop {r7, pc}
+;
+; BE-FP16-LABEL: caller_soft_half_in_reg:
+; BE-FP16: @ %bb.0: @ %entry
+; BE-FP16-NEXT: .save {r7, lr}
+; BE-FP16-NEXT: push {r7, lr}
+; BE-FP16-NEXT: mov.w r0, #15360
+; BE-FP16-NEXT: bl callee_soft_half_in_reg
+; BE-FP16-NEXT: pop {r7, pc}
+entry:
+ %ret = call arm_aapcscc half @callee_soft_half_in_reg(half 1.0)
+ ret void
+}
+
+define arm_aapcscc half @callee_soft_half_on_stack(float %r0, float %r1, float %r2, float %r3, half %f) {
+; LE-LABEL: callee_soft_half_on_stack:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: ldr r0, [sp]
+; LE-NEXT: bx lr
+;
+; BE-LABEL: callee_soft_half_on_stack:
+; BE: @ %bb.0: @ %entry
+; BE-NEXT: ldr r0, [sp]
+; BE-NEXT: bx lr
+;
+; LE-FP16-LABEL: callee_soft_half_on_stack:
+; LE-FP16: @ %bb.0: @ %entry
+; LE-FP16-NEXT: vldr.16 s0, [sp]
+; LE-FP16-NEXT: vmov r0, s0
+; LE-FP16-NEXT: bx lr
+;
+; BE-FP16-LABEL: callee_soft_half_on_stack:
+; BE-FP16: @ %bb.0: @ %entry
+; BE-FP16-NEXT: vldr.16 s0, [sp, #2]
+; BE-FP16-NEXT: vmov r0, s0
+; BE-FP16-NEXT: bx lr
+entry:
+ ret half %f
+}
+
+define void @caller_soft_half_on_stack() {
+; LE-LABEL: caller_soft_half_on_stack:
+; LE: @ %bb.0: @ %entry
+; LE-NEXT: .save {r7, lr}
+; LE-NEXT: push {r7, lr}
+; LE-NEXT: .pad #8
+; LE-NEXT: sub sp, #8
+; LE-NEXT: mov.w r0, #15360
+; LE-NEXT: str r0, [sp]
+; LE-NEXT: bl callee_soft_half_on_stack
+; LE-NEXT: add sp, #8
+; LE-NEXT: pop {r7, pc}
+;
+; BE-LA...
[truncated]
|
|
✅ With the latest revision this PR passed the undef deprecator. |
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
| SDValue Load = | ||
| DCI.DAG.getLoad(N->getValueType(0), SDLoc(N), LN0->getChain(), | ||
| LN0->getBasePtr(), LN0->getMemOperand()); | ||
| (LN0->getMemoryVT() == MVT::i16 || LN0->getMemoryVT() == MVT::i32)) { |
There was a problem hiding this comment.
Maybe instead of hacking this, make the call lowering code generate a 16-bit load?
If you are going to mess with the width of the load, you need an LN0->isSimple() check.
| ; RUN: llc -mtriple=armv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16 | FileCheck %s --check-prefix=LE | ||
| ; RUN: llc -mtriple=armebv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16 | FileCheck %s --check-prefix=BE | ||
| ; RUN: llc -mtriple=armv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16,+fullfp16 | FileCheck %s --check-prefix=LE-FP16 | ||
| ; RUN: llc -mtriple=armebv8m.main-none-eabi < %s -frame-pointer=none -mattr=+fp-armv8d16,+fullfp16 | FileCheck %s --check-prefix=BE-FP16 |
There was a problem hiding this comment.
Maybe we also want globalisel testing here? We currently fallback, but we need tests for when we eventually don't...
There was a problem hiding this comment.
Done, there are a few cases where we don't fall back, but do still generate correct code.
flovent
pushed a commit
to flovent/llvm-project
that referenced
this pull request
Feb 13, 2025
AAPCS32 defines the fp16 and bf16 types as being passed as if they were extended to 32 bits, with the high 16 bits being unspecified. The extension is specified as happening as-if it was done in a register, which means that for big endian targets, the actual value gets passed in the higher addressed half of the stack slot, instead of the lower addressed half as for little endian. Previously, for targets with the fp16 extension, we were passing these types as a 16 bit stack slot, which worked for little endian because every later stack slot would be 4-byte aligned leaving the 2 byte gap, but was incorrect for big endian.
joaosaffran
pushed a commit
to joaosaffran/llvm-project
that referenced
this pull request
Feb 14, 2025
AAPCS32 defines the fp16 and bf16 types as being passed as if they were extended to 32 bits, with the high 16 bits being unspecified. The extension is specified as happening as-if it was done in a register, which means that for big endian targets, the actual value gets passed in the higher addressed half of the stack slot, instead of the lower addressed half as for little endian. Previously, for targets with the fp16 extension, we were passing these types as a 16 bit stack slot, which worked for little endian because every later stack slot would be 4-byte aligned leaving the 2 byte gap, but was incorrect for big endian.
sivan-shani
pushed a commit
to sivan-shani/llvm-project
that referenced
this pull request
Feb 24, 2025
AAPCS32 defines the fp16 and bf16 types as being passed as if they were extended to 32 bits, with the high 16 bits being unspecified. The extension is specified as happening as-if it was done in a register, which means that for big endian targets, the actual value gets passed in the higher addressed half of the stack slot, instead of the lower addressed half as for little endian. Previously, for targets with the fp16 extension, we were passing these types as a 16 bit stack slot, which worked for little endian because every later stack slot would be 4-byte aligned leaving the 2 byte gap, but was incorrect for big endian.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AAPCS32 defines the fp16 and bf16 types as being passed as if they were extended to 32 bits, with the high 16 bits being unspecified. The extension is specified as happening as-if it was done in a register, which means that for big endian targets, the actual value gets passed in the higher addressed half of the stack slot, instead of the lower addressed half as for little endian. Previously, for targets with the fp16 extension, we were passing these types as a 16 bit stack slot, which worked for little endian because every later stack slot would be 4-byte aligned leaving the 2 byte gap, but was incorrect for big endian.
There was also a selection failure when passing bf16 values on the stack without the bf16 hardware extension, this patch fixes that as a side effect by initially emitting the load as i32, and optimising to a bf16 load if the type is legal.