[LLVM][CodeGen][SVE] Prefer NEON instructions when zeroing Z registers. #133929
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MacDue
reviewed
Apr 1, 2025
Comment on lines
+7717
to
7735
| let Predicates = [HasNEON] in { | ||
| def : Pat<(v2i64 immAllZerosV), (MOVIv2d_ns (i32 0))>; |
There was a problem hiding this comment.
Presumably, these other fixed-vector patterns were already not reachable in streaming-mode?
There was a problem hiding this comment.
Yes, the fixed length spat/build vector would have been legalised so that it cannot get this far. I figured since I was protecting the code for scalable vector types I may as well just protect the whole related block.
Several implementations have zero-latency instructions to zero registers. To-date no implementation has a dedicated SVE instruction but we can use the NEON equivalent because it is defined to zero bits 128..VL regardless of the immediate used. NOTE: The relevant instruction is not available in streaming mode, where the original SVE DUP instruction remains in use.
paulwalker-arm
force-pushed
the
sve-splat-zero
branch
from
fc8a77e to
c21f0ea
Compare
|
@llvm/pr-subscribers-backend-aarch64 Author: Paul Walker (paulwalker-arm) ChangesSeveral implementations have zero-latency instructions to zero registers. To-date no implementation has a dedicated SVE instruction but we can use the NEON equivalent because it is defined to zero bits 128..VL regardless of the immediate used. NOTE: The relevant instruction is not available in streaming mode, where the original SVE DUP instruction remains in use. Patch is 283.97 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/133929.diff 56 Files Affected:
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.td b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
index f291589e04c6b..a3b1ae55df028 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.td
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
@@ -7731,6 +7731,7 @@ def MOVIv2d_ns : SIMDModifiedImmVectorNoShift<1, 1, 0, 0b1110, V128,
"movi", ".2d",
[(set (v2i64 V128:$Rd), (AArch64movi_edit imm0_255:$imm8))]>;
+let Predicates = [HasNEON] in {
def : Pat<(v2i64 immAllZerosV), (MOVIv2d_ns (i32 0))>;
def : Pat<(v4i32 immAllZerosV), (MOVIv2d_ns (i32 0))>;
def : Pat<(v8i16 immAllZerosV), (MOVIv2d_ns (i32 0))>;
@@ -7740,6 +7741,23 @@ def : Pat<(v4f32 immAllZerosV), (MOVIv2d_ns (i32 0))>;
def : Pat<(v8f16 immAllZerosV), (MOVIv2d_ns (i32 0))>;
def : Pat<(v8bf16 immAllZerosV), (MOVIv2d_ns (i32 0))>;
+// Prefer NEON instructions when zeroing ZPRs because they are potentially zero-latency.
+let AddedComplexity = 5 in {
+def : Pat<(nxv2i64 (splat_vector (i64 0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv4i32 (splat_vector (i32 0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv8i16 (splat_vector (i32 0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv16i8 (splat_vector (i32 0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv2f64 (splat_vector (f64 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv2f32 (splat_vector (f32 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv4f32 (splat_vector (f32 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv2f16 (splat_vector (f16 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv4f16 (splat_vector (f16 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv8f16 (splat_vector (f16 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv2bf16 (splat_vector (bf16 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv4bf16 (splat_vector (bf16 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+def : Pat<(nxv8bf16 (splat_vector (bf16 fpimm0))), (SUBREG_TO_REG (i32 0), (MOVIv2d_ns (i32 0)), zsub)>;
+}
+
def : Pat<(v2i64 immAllOnesV), (MOVIv2d_ns (i32 255))>;
def : Pat<(v4i32 immAllOnesV), (MOVIv2d_ns (i32 255))>;
def : Pat<(v8i16 immAllOnesV), (MOVIv2d_ns (i32 255))>;
@@ -7760,6 +7778,7 @@ def : Pat<(v1i64 immAllOnesV), (EXTRACT_SUBREG (MOVIv2d_ns (i32 255)), dsub)>;
def : Pat<(v2i32 immAllOnesV), (EXTRACT_SUBREG (MOVIv2d_ns (i32 255)), dsub)>;
def : Pat<(v4i16 immAllOnesV), (EXTRACT_SUBREG (MOVIv2d_ns (i32 255)), dsub)>;
def : Pat<(v8i8 immAllOnesV), (EXTRACT_SUBREG (MOVIv2d_ns (i32 255)), dsub)>;
+}
// EDIT per word & halfword: 2s, 4h, 4s, & 8h
let isReMaterializable = 1, isAsCheapAsAMove = 1 in
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-contract.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-contract.ll
index 98f5b4c19a9b9..533e831de0df8 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-contract.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-contract.ll
@@ -50,10 +50,10 @@ entry:
define <vscale x 4 x double> @mul_add_mull(<vscale x 4 x double> %a, <vscale x 4 x double> %b, <vscale x 4 x double> %c, <vscale x 4 x double> %d) {
; CHECK-LABEL: mul_add_mull:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
-; CHECK-NEXT: mov z26.d, #0 // =0x0
-; CHECK-NEXT: mov z27.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z24.d, p0/m, z2.d, z0.d, #0
; CHECK-NEXT: fcmla z25.d, p0/m, z3.d, z1.d, #0
@@ -101,10 +101,10 @@ entry:
define <vscale x 4 x double> @mul_sub_mull(<vscale x 4 x double> %a, <vscale x 4 x double> %b, <vscale x 4 x double> %c, <vscale x 4 x double> %d) {
; CHECK-LABEL: mul_sub_mull:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
-; CHECK-NEXT: mov z26.d, #0 // =0x0
-; CHECK-NEXT: mov z27.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z24.d, p0/m, z2.d, z0.d, #0
; CHECK-NEXT: fcmla z25.d, p0/m, z3.d, z1.d, #0
@@ -152,10 +152,10 @@ entry:
define <vscale x 4 x double> @mul_conj_mull(<vscale x 4 x double> %a, <vscale x 4 x double> %b, <vscale x 4 x double> %c, <vscale x 4 x double> %d) {
; CHECK-LABEL: mul_conj_mull:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
-; CHECK-NEXT: mov z26.d, #0 // =0x0
-; CHECK-NEXT: mov z27.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z24.d, p0/m, z2.d, z0.d, #0
; CHECK-NEXT: fcmla z25.d, p0/m, z3.d, z1.d, #0
@@ -204,7 +204,7 @@ define <vscale x 4 x double> @mul_add_rot_mull(<vscale x 4 x double> %a, <vscale
; CHECK-LABEL: mul_add_rot_mull:
; CHECK: // %bb.0: // %entry
; CHECK-NEXT: uzp2 z24.d, z4.d, z5.d
-; CHECK-NEXT: mov z25.d, #0 // =0x0
+; CHECK-NEXT: movi v25.2d, #0000000000000000
; CHECK-NEXT: uzp1 z4.d, z4.d, z5.d
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: mov z26.d, z24.d
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-fast.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-fast.ll
index 2fc91125bc0ac..1eed9722f57be 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-fast.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-add-mull-scalable-fast.ll
@@ -41,8 +41,8 @@ entry:
define <vscale x 4 x double> @mul_add_mull(<vscale x 4 x double> %a, <vscale x 4 x double> %b, <vscale x 4 x double> %c, <vscale x 4 x double> %d) {
; CHECK-LABEL: mul_add_mull:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z25.d, p0/m, z6.d, z4.d, #0
; CHECK-NEXT: fcmla z24.d, p0/m, z7.d, z5.d, #0
@@ -90,8 +90,8 @@ entry:
define <vscale x 4 x double> @mul_sub_mull(<vscale x 4 x double> %a, <vscale x 4 x double> %b, <vscale x 4 x double> %c, <vscale x 4 x double> %d) {
; CHECK-LABEL: mul_sub_mull:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z25.d, p0/m, z6.d, z4.d, #270
; CHECK-NEXT: fcmla z24.d, p0/m, z7.d, z5.d, #270
@@ -139,8 +139,8 @@ entry:
define <vscale x 4 x double> @mul_conj_mull(<vscale x 4 x double> %a, <vscale x 4 x double> %b, <vscale x 4 x double> %c, <vscale x 4 x double> %d) {
; CHECK-LABEL: mul_conj_mull:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z25.d, p0/m, z0.d, z2.d, #0
; CHECK-NEXT: fcmla z24.d, p0/m, z1.d, z3.d, #0
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-f16-mul-scalable.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-f16-mul-scalable.ll
index 80934d2cb98c2..a7442cae84c2d 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-f16-mul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-f16-mul-scalable.ll
@@ -46,7 +46,7 @@ entry:
define <vscale x 8 x half> @complex_mul_v8f16(<vscale x 8 x half> %a, <vscale x 8 x half> %b) {
; CHECK-LABEL: complex_mul_v8f16:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z2.h, #0 // =0x0
+; CHECK-NEXT: movi v2.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.h
; CHECK-NEXT: fcmla z2.h, p0/m, z1.h, z0.h, #0
; CHECK-NEXT: fcmla z2.h, p0/m, z1.h, z0.h, #90
@@ -72,8 +72,8 @@ entry:
define <vscale x 16 x half> @complex_mul_v16f16(<vscale x 16 x half> %a, <vscale x 16 x half> %b) {
; CHECK-LABEL: complex_mul_v16f16:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z4.h, #0 // =0x0
-; CHECK-NEXT: mov z5.h, #0 // =0x0
+; CHECK-NEXT: movi v4.2d, #0000000000000000
+; CHECK-NEXT: movi v5.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.h
; CHECK-NEXT: fcmla z5.h, p0/m, z2.h, z0.h, #0
; CHECK-NEXT: fcmla z4.h, p0/m, z3.h, z1.h, #0
@@ -103,10 +103,10 @@ entry:
define <vscale x 32 x half> @complex_mul_v32f16(<vscale x 32 x half> %a, <vscale x 32 x half> %b) {
; CHECK-LABEL: complex_mul_v32f16:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.h, #0 // =0x0
-; CHECK-NEXT: mov z25.h, #0 // =0x0
-; CHECK-NEXT: mov z26.h, #0 // =0x0
-; CHECK-NEXT: mov z27.h, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.h
; CHECK-NEXT: fcmla z24.h, p0/m, z4.h, z0.h, #0
; CHECK-NEXT: fcmla z25.h, p0/m, z5.h, z1.h, #0
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-f32-mul-scalable.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-f32-mul-scalable.ll
index 874b5b538f1fd..3cad74b7f5fc6 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-f32-mul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-f32-mul-scalable.ll
@@ -7,7 +7,7 @@ target triple = "aarch64"
define <vscale x 4 x float> @complex_mul_v4f32(<vscale x 4 x float> %a, <vscale x 4 x float> %b) {
; CHECK-LABEL: complex_mul_v4f32:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z2.s, #0 // =0x0
+; CHECK-NEXT: movi v2.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.s
; CHECK-NEXT: fcmla z2.s, p0/m, z1.s, z0.s, #0
; CHECK-NEXT: fcmla z2.s, p0/m, z1.s, z0.s, #90
@@ -34,8 +34,8 @@ entry:
define <vscale x 8 x float> @complex_mul_v8f32(<vscale x 8 x float> %a, <vscale x 8 x float> %b) {
; CHECK-LABEL: complex_mul_v8f32:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z4.s, #0 // =0x0
-; CHECK-NEXT: mov z5.s, #0 // =0x0
+; CHECK-NEXT: movi v4.2d, #0000000000000000
+; CHECK-NEXT: movi v5.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.s
; CHECK-NEXT: fcmla z5.s, p0/m, z2.s, z0.s, #0
; CHECK-NEXT: fcmla z4.s, p0/m, z3.s, z1.s, #0
@@ -65,10 +65,10 @@ entry:
define <vscale x 16 x float> @complex_mul_v16f32(<vscale x 16 x float> %a, <vscale x 16 x float> %b) {
; CHECK-LABEL: complex_mul_v16f32:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.s, #0 // =0x0
-; CHECK-NEXT: mov z25.s, #0 // =0x0
-; CHECK-NEXT: mov z26.s, #0 // =0x0
-; CHECK-NEXT: mov z27.s, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.s
; CHECK-NEXT: fcmla z24.s, p0/m, z4.s, z0.s, #0
; CHECK-NEXT: fcmla z25.s, p0/m, z5.s, z1.s, #0
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-f64-mul-scalable.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-f64-mul-scalable.ll
index c9a092f52f159..e3d99fa457bbc 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-f64-mul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-f64-mul-scalable.ll
@@ -7,7 +7,7 @@ target triple = "aarch64"
define <vscale x 2 x double> @complex_mul_v2f64(<vscale x 2 x double> %a, <vscale x 2 x double> %b) {
; CHECK-LABEL: complex_mul_v2f64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z2.d, #0 // =0x0
+; CHECK-NEXT: movi v2.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z2.d, p0/m, z1.d, z0.d, #0
; CHECK-NEXT: fcmla z2.d, p0/m, z1.d, z0.d, #90
@@ -34,8 +34,8 @@ entry:
define <vscale x 4 x double> @complex_mul_v4f64(<vscale x 4 x double> %a, <vscale x 4 x double> %b) {
; CHECK-LABEL: complex_mul_v4f64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z4.d, #0 // =0x0
-; CHECK-NEXT: mov z5.d, #0 // =0x0
+; CHECK-NEXT: movi v4.2d, #0000000000000000
+; CHECK-NEXT: movi v5.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z5.d, p0/m, z2.d, z0.d, #0
; CHECK-NEXT: fcmla z4.d, p0/m, z3.d, z1.d, #0
@@ -65,10 +65,10 @@ entry:
define <vscale x 8 x double> @complex_mul_v8f64(<vscale x 8 x double> %a, <vscale x 8 x double> %b) {
; CHECK-LABEL: complex_mul_v8f64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
-; CHECK-NEXT: mov z26.d, #0 // =0x0
-; CHECK-NEXT: mov z27.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: ptrue p0.d
; CHECK-NEXT: fcmla z24.d, p0/m, z4.d, z0.d, #0
; CHECK-NEXT: fcmla z25.d, p0/m, z5.d, z1.d, #0
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-i16-mul-scalable.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-i16-mul-scalable.ll
index 58a0809ee093f..061fd07489284 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-i16-mul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-i16-mul-scalable.ll
@@ -46,7 +46,7 @@ entry:
define <vscale x 8 x i16> @complex_mul_v8i16(<vscale x 8 x i16> %a, <vscale x 8 x i16> %b) {
; CHECK-LABEL: complex_mul_v8i16:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z2.h, #0 // =0x0
+; CHECK-NEXT: movi v2.2d, #0000000000000000
; CHECK-NEXT: cmla z2.h, z1.h, z0.h, #0
; CHECK-NEXT: cmla z2.h, z1.h, z0.h, #90
; CHECK-NEXT: mov z0.d, z2.d
@@ -71,8 +71,8 @@ entry:
define <vscale x 16 x i16> @complex_mul_v16i16(<vscale x 16 x i16> %a, <vscale x 16 x i16> %b) {
; CHECK-LABEL: complex_mul_v16i16:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z4.h, #0 // =0x0
-; CHECK-NEXT: mov z5.h, #0 // =0x0
+; CHECK-NEXT: movi v4.2d, #0000000000000000
+; CHECK-NEXT: movi v5.2d, #0000000000000000
; CHECK-NEXT: cmla z5.h, z2.h, z0.h, #0
; CHECK-NEXT: cmla z4.h, z3.h, z1.h, #0
; CHECK-NEXT: cmla z5.h, z2.h, z0.h, #90
@@ -101,10 +101,10 @@ entry:
define <vscale x 32 x i16> @complex_mul_v32i16(<vscale x 32 x i16> %a, <vscale x 32 x i16> %b) {
; CHECK-LABEL: complex_mul_v32i16:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.h, #0 // =0x0
-; CHECK-NEXT: mov z25.h, #0 // =0x0
-; CHECK-NEXT: mov z26.h, #0 // =0x0
-; CHECK-NEXT: mov z27.h, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: cmla z24.h, z4.h, z0.h, #0
; CHECK-NEXT: cmla z25.h, z5.h, z1.h, #0
; CHECK-NEXT: cmla z27.h, z6.h, z2.h, #0
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-i32-mul-scalable.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-i32-mul-scalable.ll
index 0958c60ed7cb0..52caa3279b927 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-i32-mul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-i32-mul-scalable.ll
@@ -7,7 +7,7 @@ target triple = "aarch64"
define <vscale x 4 x i32> @complex_mul_v4i32(<vscale x 4 x i32> %a, <vscale x 4 x i32> %b) {
; CHECK-LABEL: complex_mul_v4i32:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z2.s, #0 // =0x0
+; CHECK-NEXT: movi v2.2d, #0000000000000000
; CHECK-NEXT: cmla z2.s, z1.s, z0.s, #0
; CHECK-NEXT: cmla z2.s, z1.s, z0.s, #90
; CHECK-NEXT: mov z0.d, z2.d
@@ -33,8 +33,8 @@ entry:
define <vscale x 8 x i32> @complex_mul_v8i32(<vscale x 8 x i32> %a, <vscale x 8 x i32> %b) {
; CHECK-LABEL: complex_mul_v8i32:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z4.s, #0 // =0x0
-; CHECK-NEXT: mov z5.s, #0 // =0x0
+; CHECK-NEXT: movi v4.2d, #0000000000000000
+; CHECK-NEXT: movi v5.2d, #0000000000000000
; CHECK-NEXT: cmla z5.s, z2.s, z0.s, #0
; CHECK-NEXT: cmla z4.s, z3.s, z1.s, #0
; CHECK-NEXT: cmla z5.s, z2.s, z0.s, #90
@@ -63,10 +63,10 @@ entry:
define <vscale x 16 x i32> @complex_mul_v16i32(<vscale x 16 x i32> %a, <vscale x 16 x i32> %b) {
; CHECK-LABEL: complex_mul_v16i32:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.s, #0 // =0x0
-; CHECK-NEXT: mov z25.s, #0 // =0x0
-; CHECK-NEXT: mov z26.s, #0 // =0x0
-; CHECK-NEXT: mov z27.s, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: cmla z24.s, z4.s, z0.s, #0
; CHECK-NEXT: cmla z25.s, z5.s, z1.s, #0
; CHECK-NEXT: cmla z27.s, z6.s, z2.s, #0
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-i64-mul-scalable.ll b/llvm/test/CodeGen/AArch64/complex-deinterleaving-i64-mul-scalable.ll
index 30c06838c81bc..bdc21e7828277 100644
--- a/llvm/test/CodeGen/AArch64/complex-deinterleaving-i64-mul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/complex-deinterleaving-i64-mul-scalable.ll
@@ -7,7 +7,7 @@ target triple = "aarch64"
define <vscale x 2 x i64> @complex_mul_v2i64(<vscale x 2 x i64> %a, <vscale x 2 x i64> %b) {
; CHECK-LABEL: complex_mul_v2i64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z2.d, #0 // =0x0
+; CHECK-NEXT: movi v2.2d, #0000000000000000
; CHECK-NEXT: cmla z2.d, z1.d, z0.d, #0
; CHECK-NEXT: cmla z2.d, z1.d, z0.d, #90
; CHECK-NEXT: mov z0.d, z2.d
@@ -33,8 +33,8 @@ entry:
define <vscale x 4 x i64> @complex_mul_v4i64(<vscale x 4 x i64> %a, <vscale x 4 x i64> %b) {
; CHECK-LABEL: complex_mul_v4i64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z4.d, #0 // =0x0
-; CHECK-NEXT: mov z5.d, #0 // =0x0
+; CHECK-NEXT: movi v4.2d, #0000000000000000
+; CHECK-NEXT: movi v5.2d, #0000000000000000
; CHECK-NEXT: cmla z5.d, z2.d, z0.d, #0
; CHECK-NEXT: cmla z4.d, z3.d, z1.d, #0
; CHECK-NEXT: cmla z5.d, z2.d, z0.d, #90
@@ -63,10 +63,10 @@ entry:
define <vscale x 8 x i64> @complex_mul_v8i64(<vscale x 8 x i64> %a, <vscale x 8 x i64> %b) {
; CHECK-LABEL: complex_mul_v8i64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
-; CHECK-NEXT: mov z26.d, #0 // =0x0
-; CHECK-NEXT: mov z27.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: cmla z24.d, z4.d, z0.d, #0
; CHECK-NEXT: cmla z25.d, z5.d, z1.d, #0
; CHECK-NEXT: cmla z27.d, z6.d, z2.d, #0
@@ -101,10 +101,10 @@ entry:
define <vscale x 8 x i64> @complex_minus_mul_v8i64(<vscale x 8 x i64> %a, <vscale x 8 x i64> %b) {
; CHECK-LABEL: complex_minus_mul_v8i64:
; CHECK: // %bb.0: // %entry
-; CHECK-NEXT: mov z24.d, #0 // =0x0
-; CHECK-NEXT: mov z25.d, #0 // =0x0
-; CHECK-NEXT: mov z26.d, #0 // =0x0
-; CHECK-NEXT: mov z27.d, #0 // =0x0
+; CHECK-NEXT: movi v24.2d, #0000000000000000
+; CHECK-NEXT: movi v25.2d, #0000000000000000
+; CHECK-NEXT: movi v26.2d, #0000000000000000
+; CHECK-NEXT: movi v27.2d, #0000000000000000
; CHECK-NEXT: cmla z24.d, z4.d, z0.d, #270
; CHECK-NEXT: cmla z25.d, z5.d, z1.d, #270
; CHECK-NEXT: cmla z27.d, z6.d, z2.d, #270
diff --git a/llvm/test/CodeGen/AArch64/complex-deinterleaving-reductions-p...
[truncated]
|
sdesmalen-arm
approved these changes
Apr 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Several implementations have zero-latency instructions to zero registers. To-date no implementation has a dedicated SVE instruction but we can use the NEON equivalent because it is defined to zero bits 128..VL regardless of the immediate used.
NOTE: The relevant instruction is not available in streaming mode, where the original SVE DUP instruction remains in use.