CODEWITHSUNDEEP
linuxnetwork-programminglinux-device-driver
Amit
Amit

Reputation: 187

Linux - Transmit Packet Steering (XPS) issue

I have a network driver (driver interface eth1) with 12 TX queues and running under 12 CPU cores.

I am using Amazon Linux 2: 4.14.243-185.433.amzn2.x86_64

I am trying to use XPS to steer TX packets for each queue to a single core (according to https://www.kernel.org/doc/html/v5.10/networking/scaling.html)

To config the XPS, I run the following:

echo 1 > /sys/class/net/eth1/queues/tx-0/xps_cpus
echo 2 > /sys/class/net/eth1/queues/tx-1/xps_cpus
echo 4 > /sys/class/net/eth1/queues/tx-2/xps_cpus
echo 8 > /sys/class/net/eth1/queues/tx-3/xps_cpus
echo 10 > /sys/class/net/eth1/queues/tx-4/xps_cpus
echo 20 > /sys/class/net/eth1/queues/tx-5/xps_cpus
echo 40 > /sys/class/net/eth1/queues/tx-6/xps_cpus
echo 80 > /sys/class/net/eth1/queues/tx-7/xps_cpus
echo 100 > /sys/class/net/eth1/queues/tx-8/xps_cpus
echo 200 > /sys/class/net/eth1/queues/tx-9/xps_cpus
echo 400 > /sys/class/net/eth1/queues/tx-10/xps_cpus
echo 800 > /sys/class/net/eth1/queues/tx-11/xps_cpus

Using iperf2, when sending single flow (single queue) it doesn't seem to use the xps configuration above, every run, a different CPU will start and transmit the flow even if source/dest IP and ports stay the same. it also doesn't seem to choose the same queue every time, any reason why? command:

iperf -c 1.2.3.4 -p 5001 -B 5.6.7.8:5002

Tried also to bind the iperf to specific core and it still didn't work:

numactl --physcpubind=0 iperf -c 1.2.3.4 -p 5001 -B 5.6.7.8:5002

Validated that the driver is calling __netdev_pick_tx to choose the queue index: https://elixir.bootlin.com/linux/v4....re/dev.c#L3196

Any other thing needs to be done to enable XPS?

how to check if CONFIG_XPS is enabled?

Any help will be appreciated.

Thanks

Upvotes: 0

Views: 2467

Answers (2)

Manish Chopra
Manish Chopra

Reputation: 11

Regarding your question on why it chooses a different queue every time, it may be due to a connection hash which is different every time. Generally for fixed known connections (src ip, dst ip, src port, dst port) it will always be a unique queue, but iperf gets launched every time with random TCP/UDP ports. That's why the connection hash changes, probably to pick up different queue. With netperf, you can always run with fixed ports using the -- -P option.

netperf -H 20.0.0.18 -t TCP_STREAM -- -P 6660,5550

Upvotes: 1

Manish Chopra
Manish Chopra

Reputation: 11

I believe numactl or taskset affinities are not governed by iperf and friends for some reason, I also faced this problem many times although you can try below iperf3 with -A option. It may affinitize iperf client on this core.

iperf3 -c 192.168.40.100 -A5,5 -t 23456 -P 1

It's possible that the core who is running the connection is not from the cores you have set for XPS. XPS is basically for a given core to decide which queue to be selected if it's mask is found in one of the queue's xps_cpus and if that core (which is unknown at the moment who is running that connection) is not in xps_cpus queue config then it could select any random queue based on packet/connection hash.

if you need to find out which core is running the connection, then let's say at run time the queue 0 is used (based on ethtool -S ethx counter) for transmit, try one by one setting all the cores in system to queue 1 (it could be any queue other than the queue 0) xps_cpus and see for which core set the traffic got switched on the queue 1 (based on ethtool -S ethx again), that will be the core who is running your connection actually.

HTH Thanks, Manish

Upvotes: 0

Related Questions