net: traffic blocks when running ingress UDP performance tests with very large buffers

[raduiliescu]

On a m5.metal host running Amazon Linux, create a microVM with 8 cores and one network interface using xenial rootfs
Use the recommended network setup for host and guest - https://github.com/firecracker-microvm/firecracker/blob/master/docs/network-setup.md

Increase RX buffers in VM by 100 times
echo 21299200 > /proc/sys/net/core/rmem_max
echo 21299200 > /proc/sys/net/core/rmem_default

Start an iperf3 server on VM - iperf3 -p 11111 -s 172.16.0.2 -D
Start iperf3 udp client on host: iperf3 -p 11111 -u -c 172.16.0.2 -V -b 1000000000 -t 10000

After some time all networking of the VM will stop working. This happens with latest code as well as with latest version.

[ 4] 36.00-37.00 sec 119 MBytes 1.00 Gbits/sec 86332
[ 4] 37.00-38.00 sec 119 MBytes 999 Mbits/sec 86263
[ 4] 38.00-39.00 sec 119 MBytes 1.00 Gbits/sec 86326
[ 4] 39.00-40.00 sec 59.7 MBytes 501 Mbits/sec 43216
[ 4] 40.00-41.00 sec 0.00 Bytes 0.00 bits/sec 0
[ 4] 41.00-42.00 sec 0.00 Bytes 0.00 bits/sec 0
[ 4] 42.00-43.00 sec 66.5 KBytes 544 Kbits/sec 47
[ 4] 43.00-44.00 sec 0.00 Bytes 0.00 bits/sec 0

Script to start the VM:

#!/bin/sh

arch=`uname -m`
kernel_path=$(pwd)"/hello-vmlinux.bin"

if [ ${arch} = "x86_64" ]; then
	curl --unix-socket /tmp/firecracker.socket -i \
        -X PUT 'http://localhost/boot-source'   \
        -H 'Accept: application/json'           \
        -H 'Content-Type: application/json'     \
        -d "{
              \"kernel_image_path\": \"${kernel_path}\",
              \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off\"
         }"
elif [ ${arch} = "aarch64" ]; then
    curl --unix-socket /tmp/firecracker.socket -i \
        -X PUT 'http://localhost/boot-source'   \
        -H 'Accept: application/json'           \
        -H 'Content-Type: application/json'     \
        -d "{
              \"kernel_image_path\": \"${kernel_path}\",
              \"boot_args\": \"keep_bootcon console=ttyS0 reboot=k panic=1 pci=off\"
         }"
else
    echo "Cannot run firecracker on $arch architecture!"
    exit 1
fi

rootfs_path=$(pwd)"/xenial.rootfs.ext4"
#rootfs_path=$(pwd)"/hello-rootfs.ext4"
curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/drives/rootfs' \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d "{
          \"drive_id\": \"rootfs\",
          \"path_on_host\": \"${rootfs_path}\",
          \"is_root_device\": true,
          \"is_read_only\": false
     }"

curl --unix-socket /tmp/firecracker.socket -i  \
    -X PUT 'http://localhost/machine-config' \
    -H 'Accept: application/json'            \
    -H 'Content-Type: application/json'      \
    -d '{
        "vcpu_count": 8,
        "mem_size_mib": 4096,
        "ht_enabled": false
    }'

curl --unix-socket /tmp/firecracker.socket -i \
     -X PUT 'http://localhost/network-interfaces/eth0' \
     -H 'Accept: application/json' \
     -H 'Content-Type: application/json' \
       -d '{
      "iface_id": "eth0",
      "guest_mac": "AA:FC:00:00:00:01",
      "host_dev_name": "tap0"
    }'

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/actions'       \
    -H  'Accept: application/json'          \
    -H  'Content-Type: application/json'    \
    -d '{
        "action_type": "InstanceStart"
     }'

[ioanachirca]

Duplicate of #2057