CODEWITHSUNDEEP
network-programmingkernelvirtualizationkvm
Houcheng
Houcheng

Reputation: 2884

The QEMU-KVM guest OS does not respect MTU setting

I have 3 QEMU-KVM VM running on Ubuntu host. Each VM has one virt-io nic and connected by a br0 bridge in host OS. Every NIC's MTU is configured to 1500. This platform runs a message queue based application: one VM as producer, one VM as message queue server and last VM as consumer.

My problem is: When guest VMs become busy, the message queue server VM may ocasually send packet in size 2k, 4k or 5k. It exceed the NIC's MTU value, 1500!!! As I know the Linux IP stack should cut the IP framgments smaller than size of MTU before sending out. Why this VM sends large packet when busy?

Below is the platform's architecture:

[w2: celery caller (message queue producer) ] 
     | (messages in TCP)
     V
[w3: rabbitmq server] 
     | (messages in TCP)
     V
[w4: celeryd (message queue consumer)]

Below is screen captured at w3, MTU is 1500 and sends packet with size > 2000 screen captured at w3

Below is screen captured at w4, MTU is 1500 and receives packet with size > 2000 enter image description here

Upvotes: 1

Views: 6385

Answers (1)

Houcheng
Houcheng

Reputation: 2884

Thanks all, I found the answer now.

The virtio-NIC in guest OS supports TCP Segmentation Offload (TSO) and it is enabled by default. The NIC's TSO feature is TCP layer does not do the frame segmentation, and the segmentation is perform offline by NIC driver or hardware. The TSO is for enhance performance when sending large packet and offload the worknig of TCP sender program, the NIC driver will receive the large packet in this mode and segment it into smaller payload, IP header and TCP header. The benefit of this offload processing is driver can well use the hardware or reuse the header skb buffer during segmentation.

Below is driver code that supports TSO feature,

Atheros NIC driver source

On its sending function, it calls atl1c_tso_csum() that checks transmitting buffer's flags & SKB_GSO_TCPV4. if the buffer contains SKB_GSO_TCPV4 flag, segment it, add ip header and add tcp header.

The virtio NIC support TSO feature, however, as virtio NIC packets is sending to host OS. Its driver's tx funciton just send out the whole packet and achieve higher network bandwidth.

Note: after turning off TSO flag in guest VM, no large packets is seen on tcpdump. here is the commands:

$ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
$ethtool -K eth0 tso off
$

reference 2: https://blogs.gnome.org/markmc/category/virtio/

Upvotes: 3

Related Questions