Skip to content

Commit 51cf49f

Browse files
kuba-mookuba-moo
committed
Merge branch 'selftests-fixes-for-kernel-ci'
Petr Machata says: ==================== selftests: Fixes for kernel CI As discussed on the bi-weekly call on Jan 30, and in mailing around kernel CI effort, some changes are desirable in the suite of forwarding selftests the better to work with the CI tooling. Namely: - The forwarding selftests use a configuration file where names of interfaces are defined and various variables can be overridden. There is also forwarding.config.sample that users can use as a template to refer to when creating the config file. What happens a fair bit is that users either do not know about this at all, or simply forget, and are confused by cryptic failures about interfaces that cannot be created. In patches #1 - #3 have lib.sh just be the single source of truth with regards to which variables exist. That includes the topology variables which were previously only in the sample file, and any "tweak variables", such as what tools to use, sleep times, etc. forwarding.config.sample then becomes just a placeholder with a couple examples. Unless specific HW should be exercised, or specific tools used, the defaults are usually just fine. - Several net/forwarding/ selftests (and one net/ one) cannot be run on veth pairs, they need an actual HW interface to run on. They are generic in the sense that any capable HW should pass them, which is why they have been put to net/forwarding/ as opposed to drivers/net/, but they do not generalize to veth. The fact that these tests are in net/forwarding/, but still complaining when run, is confusing. In patches #4 - #6 move these tests to a new directory drivers/net/hw. - The following patches extend the codebase to handle well test results other than pass and fail. Patch #7 is preparatory. It converts several log_test_skip to XFAIL, so that tests do not spuriously end up returning non-0 when they are not supposed to. In patches #8 - #10, introduce some missing ksft constants, then support having those constants in RET, and then finally in EXIT_STATUS. - The traffic scheduler tests generate a large amount of network traffic to test the behavior of the scheduler. This demands a relatively high-performance computer. On slow machines, such as with a debugging kernel, the test would spuriously fail. It can still be useful to "go through the motions" though, to possibly catch bugs in setup of the scheduler graph and passing packets around. Thus we still want to run the tests, just with lowered demands. To that end, in patches #11 - #12, introduce an environment variable KSFT_MACHINE_SLOW, with obvious meaning. Tests can then make checks more lenient, such as mark failures as XFAIL. A helper, xfail_on_slow, is provided to mark performance-sensitive parts of the selftest. - In patch #13, use a similar mechanism to mark a NH group stats selftest to XFAIL HW stats tests when run on VETH pairs. - All these changes complicate the hitherto straightforward logging and checking logic, so in patch #14, add a selftest that checks this functionality in lib.sh. v1 (vs. an RFC circulated through linux-kselftest): - Patch #9: - Clarify intended usage by s/set_ret/ret_set_ksft_status/, s/nret/ksft_status/ ==================== Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Jakub Kicinski <[email protected]>
2 parents 5e47fbe + 8ff2d7a commit 51cf49f

28 files changed

+565
-171
lines changed

‎tools/testing/selftests/drivers/net/hw/Makefile

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# SPDX-License-Identifier: GPL-2.0+ OR MIT
2+
3+
TEST_PROGS = \
4+
devlink_port_split.py \
5+
ethtool.sh \
6+
ethtool_extended_state.sh \
7+
ethtool_mm.sh \
8+
ethtool_rmon.sh \
9+
hw_stats_l3.sh \
10+
hw_stats_l3_gre.sh \
11+
loopback.sh \
12+
#
13+
14+
TEST_FILES := \
15+
ethtool_lib.sh \
16+
#
17+
18+
TEST_INCLUDES := \
19+
../../../net/lib.sh \
20+
../../../net/forwarding/lib.sh \
21+
../../../net/forwarding/ipip_lib.sh \
22+
../../../net/forwarding/tc_common.sh \
23+
#
24+
25+
include ../../../lib.mk

‎tools/testing/selftests/net/devlink_port_split.py renamed to ‎tools/testing/selftests/drivers/net/hw/devlink_port_split.py

File renamed without changes.

‎tools/testing/selftests/net/forwarding/ethtool.sh renamed to ‎tools/testing/selftests/drivers/net/hw/ethtool.sh

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ ALL_TESTS="
1010
different_speeds_autoneg_on
1111
"
1212
NUM_NETIFS=2
13-
source lib.sh
13+
lib_dir=$(dirname "$0")
14+
source "$lib_dir"/../../../net/forwarding/lib.sh
1415
source ethtool_lib.sh
1516

1617
h1_create()
@@ -286,8 +287,6 @@ different_speeds_autoneg_on()
286287
ethtool -s $h1 autoneg on
287288
}
288289

289-
skip_on_veth
290-
291290
trap cleanup EXIT
292291

293292
setup_prepare

‎tools/testing/selftests/net/forwarding/ethtool_extended_state.sh renamed to ‎tools/testing/selftests/drivers/net/hw/ethtool_extended_state.sh

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@ ALL_TESTS="
88
"
99

1010
NUM_NETIFS=2
11-
source lib.sh
11+
lib_dir=$(dirname "$0")
12+
source "$lib_dir"/../../../net/forwarding/lib.sh
1213
source ethtool_lib.sh
1314

1415
TIMEOUT=$((WAIT_TIMEOUT * 1000)) # ms
@@ -108,8 +109,6 @@ no_cable()
108109
ip link set dev $swp3 down
109110
}
110111

111-
skip_on_veth
112-
113112
setup_prepare
114113

115114
tests_run

‎tools/testing/selftests/net/forwarding/ethtool_lib.sh renamed to ‎tools/testing/selftests/drivers/net/hw/ethtool_lib.sh

File renamed without changes.

‎tools/testing/selftests/net/forwarding/ethtool_mm.sh renamed to ‎tools/testing/selftests/drivers/net/hw/ethtool_mm.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ ALL_TESTS="
1414
NUM_NETIFS=2
1515
REQUIRE_MZ=no
1616
PREEMPTIBLE_PRIO=0
17-
source lib.sh
17+
lib_dir=$(dirname "$0")
18+
source "$lib_dir"/../../../net/forwarding/lib.sh
1819

1920
traffic_test()
2021
{

‎tools/testing/selftests/net/forwarding/ethtool_rmon.sh renamed to ‎tools/testing/selftests/drivers/net/hw/ethtool_rmon.sh

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ ALL_TESTS="
77
"
88

99
NUM_NETIFS=2
10-
source lib.sh
10+
lib_dir=$(dirname "$0")
11+
source "$lib_dir"/../../../net/forwarding/lib.sh
1112

1213
ETH_FCS_LEN=4
1314
ETH_HLEN=$((6+6+2))
@@ -78,7 +79,7 @@ rmon_histogram()
7879

7980
for if in $iface $neigh; do
8081
if ! ensure_mtu $if ${bucket[0]}; then
81-
log_test_skip "$if does not support the required MTU for $step"
82+
log_test_xfail "$if does not support the required MTU for $step"
8283
return
8384
fi
8485
done
@@ -93,7 +94,7 @@ rmon_histogram()
9394
jq -r ".[0].rmon[\"${set}-pktsNtoM\"][]|[.low, .high]|@tsv" 2>/dev/null)
9495

9596
if [ $nbuckets -eq 0 ]; then
96-
log_test_skip "$iface does not support $set histogram counters"
97+
log_test_xfail "$iface does not support $set histogram counters"
9798
return
9899
fi
99100
}

‎tools/testing/selftests/net/forwarding/hw_stats_l3.sh renamed to ‎tools/testing/selftests/drivers/net/hw/hw_stats_l3.sh

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,8 @@ ALL_TESTS="
4848
test_double_enable
4949
"
5050
NUM_NETIFS=4
51-
source lib.sh
51+
lib_dir=$(dirname "$0")
52+
source "$lib_dir"/../../../net/forwarding/lib.sh
5253

5354
h1_create()
5455
{
@@ -324,17 +325,9 @@ setup_wait
324325

325326
used=$(ip -j stats show dev $rp1.200 group offload subgroup hw_stats_info |
326327
jq '.[].info.l3_stats.used')
327-
kind=$(ip -j -d link show dev $rp1 |
328-
jq -r '.[].linkinfo.info_kind')
329-
if [[ $used != true ]]; then
330-
if [[ $kind == veth ]]; then
331-
log_test_skip "l3_stats not offloaded on veth interface"
332-
EXIT_STATUS=$ksft_skip
333-
else
334-
RET=1 log_test "l3_stats not offloaded"
335-
fi
336-
else
337-
tests_run
338-
fi
328+
[[ $used = true ]]
329+
check_err $? "hw_stats_info.used=$used"
330+
log_test "l3_stats offloaded"
331+
tests_run
339332

340333
exit $EXIT_STATUS

‎tools/testing/selftests/net/forwarding/hw_stats_l3_gre.sh renamed to ‎tools/testing/selftests/drivers/net/hw/hw_stats_l3_gre.sh

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,9 @@ ALL_TESTS="
1212
test_stats_tx
1313
"
1414
NUM_NETIFS=6
15-
source lib.sh
16-
source ipip_lib.sh
15+
lib_dir=$(dirname "$0")
16+
source "$lib_dir"/../../../net/forwarding/lib.sh
17+
source "$lib_dir"/../../../net/forwarding/ipip_lib.sh
1718

1819
setup_prepare()
1920
{
@@ -99,8 +100,6 @@ test_stats_rx()
99100
test_stats g2a rx
100101
}
101102

102-
skip_on_veth
103-
104103
trap cleanup EXIT
105104

106105
setup_prepare

‎tools/testing/selftests/net/forwarding/loopback.sh renamed to ‎tools/testing/selftests/drivers/net/hw/loopback.sh

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@ ksft_skip=4
66

77
ALL_TESTS="loopback_test"
88
NUM_NETIFS=2
9-
source tc_common.sh
10-
source lib.sh
9+
lib_dir=$(dirname "$0")
10+
source "$lib_dir"/../../../net/forwarding/tc_common.sh
11+
source "$lib_dir"/../../../net/forwarding/lib.sh
1112

1213
h1_create()
1314
{

‎tools/testing/selftests/drivers/net/hw/settings

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
timeout=0

‎tools/testing/selftests/drivers/net/mlxsw/mlxsw_lib.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ __mlxsw_only_on_spectrum()
4242
local src=$1; shift
4343

4444
if ! mlxsw_on_spectrum "$rev"; then
45-
log_test_skip $src:$caller "(Spectrum-$rev only)"
45+
log_test_xfail $src:$caller "(Spectrum-$rev only)"
4646
return 1
4747
fi
4848
}

‎tools/testing/selftests/drivers/net/mlxsw/spectrum-2/resource_scale.sh

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,6 @@ for current_test in ${TESTS:-$ALL_TESTS}; do
4747
RET=0
4848
target=$(${current_test}_get_target "$should_fail")
4949
if ((target == 0)); then
50-
log_test_skip "'$current_test' should_fail=$should_fail test"
5150
continue
5251
fi
5352

‎tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,6 @@ for current_test in ${TESTS:-$ALL_TESTS}; do
5252
RET=0
5353
target=$(${current_test}_get_target "$should_fail")
5454
if ((target == 0)); then
55-
log_test_skip "'$current_test' [$profile] should_fail=$should_fail test"
5655
continue
5756
fi
5857
${current_test}_setup_prepare

‎tools/testing/selftests/net/Makefile

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ TEST_PROGS += reuseaddr_ports_exhausted.sh
2020
TEST_PROGS += txtimestamp.sh
2121
TEST_PROGS += vrf-xfrm-tests.sh
2222
TEST_PROGS += rxtimestamp.sh
23-
TEST_PROGS += devlink_port_split.py
2423
TEST_PROGS += drop_monitor_tests.sh
2524
TEST_PROGS += vrf_route_leaking.sh
2625
TEST_PROGS += bareudp.sh

‎tools/testing/selftests/net/forwarding/Makefile

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,18 +15,12 @@ TEST_PROGS = bridge_fdb_learning_limit.sh \
1515
bridge_vlan_unaware.sh \
1616
custom_multipath_hash.sh \
1717
dual_vxlan_bridge.sh \
18-
ethtool_extended_state.sh \
19-
ethtool_mm.sh \
20-
ethtool_rmon.sh \
21-
ethtool.sh \
2218
gre_custom_multipath_hash.sh \
2319
gre_inner_v4_multipath.sh \
2420
gre_inner_v6_multipath.sh \
2521
gre_multipath_nh_res.sh \
2622
gre_multipath_nh.sh \
2723
gre_multipath.sh \
28-
hw_stats_l3.sh \
29-
hw_stats_l3_gre.sh \
3024
ip6_forward_instats_vrf.sh \
3125
ip6gre_custom_multipath_hash.sh \
3226
ip6gre_flat_key.sh \
@@ -43,8 +37,8 @@ TEST_PROGS = bridge_fdb_learning_limit.sh \
4337
ipip_hier_gre_key.sh \
4438
ipip_hier_gre_keys.sh \
4539
ipip_hier_gre.sh \
40+
lib_sh_test.sh \
4641
local_termination.sh \
47-
loopback.sh \
4842
mirror_gre_bound.sh \
4943
mirror_gre_bridge_1d.sh \
5044
mirror_gre_bridge_1d_vlan.sh \
@@ -113,7 +107,6 @@ TEST_PROGS = bridge_fdb_learning_limit.sh \
113107
vxlan_symmetric.sh
114108

115109
TEST_FILES := devlink_lib.sh \
116-
ethtool_lib.sh \
117110
fib_offload_lib.sh \
118111
forwarding.config.sample \
119112
ip6gre_lib.sh \

‎tools/testing/selftests/net/forwarding/README

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,36 @@ o Checks shall be added to lib.sh for any external dependencies.
5656
o Code shall be checked using ShellCheck [1] prior to submission.
5757

5858
1. https://www.shellcheck.net/
59+
60+
Customization
61+
=============
62+
63+
The forwarding selftests framework uses a number of variables that
64+
influence its behavior and tools it invokes, and how it invokes them, in
65+
various ways. A number of these variables can be overridden. The way these
66+
overridable variables are specified is typically one of the following two
67+
syntaxes:
68+
69+
: "${VARIABLE:=default_value}"
70+
VARIABLE=${VARIABLE:=default_value}
71+
72+
Any of these variables can be overridden. Notably net/forwarding/lib.sh and
73+
net/lib.sh contain a number of overridable variables.
74+
75+
One way of overriding these variables is through the environment:
76+
77+
PAUSE_ON_FAIL=yes ./some_test.sh
78+
79+
The variable NETIFS is special. Since it is an array variable, there is no
80+
way to pass it through the environment. Its value can instead be given as
81+
consecutive arguments to the selftest:
82+
83+
./some_test.sh swp{1..8}
84+
85+
A way to customize variables in a persistent fashion is to create a file
86+
named forwarding.config in this directory. lib.sh sources the file if
87+
present, so it can contain any shell code. Typically it will contain
88+
assignments of variables whose value should be overridden.
89+
90+
forwarding.config.sample is available in the directory as an example of
91+
how forwarding.config might look.

‎tools/testing/selftests/net/forwarding/forwarding.config.sample

Lines changed: 15 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -3,51 +3,28 @@
33

44
##############################################################################
55
# Topology description. p1 looped back to p2, p3 to p4 and so on.
6-
declare -A NETIFS
76

8-
NETIFS[p1]=veth0
9-
NETIFS[p2]=veth1
10-
NETIFS[p3]=veth2
11-
NETIFS[p4]=veth3
12-
NETIFS[p5]=veth4
13-
NETIFS[p6]=veth5
14-
NETIFS[p7]=veth6
15-
NETIFS[p8]=veth7
16-
NETIFS[p9]=veth8
17-
NETIFS[p10]=veth9
7+
NETIFS=(
8+
[p1]=veth0
9+
[p2]=veth1
10+
[p3]=veth2
11+
[p4]=veth3
12+
[p5]=veth4
13+
[p6]=veth5
14+
[p7]=veth6
15+
[p8]=veth7
16+
[p9]=veth8
17+
[p10]=veth9
18+
)
1819

1920
# Port that does not have a cable connected.
2021
NETIF_NO_CABLE=eth8
2122

2223
##############################################################################
23-
# Defines
24+
# In addition to the topology-related variables, it is also possible to override
25+
# in this file other variables that net/lib.sh, net/forwarding/lib.sh or other
26+
# libraries or selftests use. E.g.:
2427

25-
# IPv4 ping utility name
26-
PING=ping
27-
# IPv6 ping utility name. Some distributions use 'ping' for IPv6.
2828
PING6=ping6
29-
# Packet generator. Some distributions use 'mz'.
3029
MZ=mausezahn
31-
# mausezahn delay between transmissions in microseconds.
32-
MZ_DELAY=0
33-
# Time to wait after interfaces participating in the test are all UP
3430
WAIT_TIME=5
35-
# Whether to pause on failure or not.
36-
PAUSE_ON_FAIL=no
37-
# Whether to pause on cleanup or not.
38-
PAUSE_ON_CLEANUP=no
39-
# Type of network interface to create
40-
NETIF_TYPE=veth
41-
# Whether to create virtual interfaces (veth) or not
42-
NETIF_CREATE=yes
43-
# Timeout (in seconds) before ping exits regardless of how many packets have
44-
# been sent or received
45-
PING_TIMEOUT=5
46-
# Minimum ageing_time (in centiseconds) supported by hardware
47-
LOW_AGEING_TIME=1000
48-
# Flag for tc match, supposed to be skip_sw/skip_hw which means do not process
49-
# filter by software/hardware
50-
TC_FLAG=skip_hw
51-
# IPv6 traceroute utility name.
52-
TROUTE6=traceroute6
53-

‎tools/testing/selftests/net/forwarding/ipip_lib.sh

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,6 @@
141141
# | $h2 + |
142142
# | 192.0.2.18/28 |
143143
# +---------------------------+
144-
source lib.sh
145144

146145
h1_create()
147146
{

0 commit comments

Comments
 (0)