Why are packet drops in UDP correlated with number of packets vs. bandwidth utilization

Question

I'm testing the speed between a client and host using iperf. In my application, I need to be able to send 2 byte UDP frames at roughly 5KHz.

Doing a normal UDP speed test I can easily get 10Mb/s:

 $iperf -uVc some_ip -b 10M
 Interval     Transfer    Bandwidth    Dropped/Sent
 0.0-10.0 sec 11.9 MBytes 10.0Mbit/sec 0 / 8504 (0%)

When I then try to mirror my application by sending 2B at 5Hz (which correlates to 80Kb/s) datagrams:

 $iperf -l 2 -uVc some_ip -b 80K

The server side says no packets made it through which I am guessing is because the counter or whatever iperf uses to track packets can't fit inside the 2B payload. Does this make sense?

As a general rule of thumb, how bad is it to send many small packets versus few large ones? Can anyone point to literature that illustrates the tradeoff between waiting to "pack" a large datagram and instantly sending out the 2B of data as soon as you get it?

To further clarify, I am interested in the penalty you pay for sending many small packets (including overhead, the packets are only about 60B) versus sending fewer, yet large packets. In my tests so far, packet drops are clearly not correlated with bandwidth use, rather they are correlated with the number of packets, which I find counter-intuitive!

EDIT:

I'm doing this on the simplest client - server setup, between two Linux PC's connected on a local network in which they are the only interfaces on the network with an Ethernet switch in between them.

Answer 1

You have to realise that your 2 bytes are payload and the networking layers will add headers in front of you payload. The IP header alone is 20 bytes, but UDP or TCP will also have their headers, and of course there is also an ethernet header.

Most likely but not garanteed the network is en ethernet network, which means the mtu is around 1500. If your packet with headers included is not that size, you are not making optimal use of the network.

IP will do fragmentation if your packet does not fit in the mtu. So assume that the mtu is 1500 and that the different layers don't add anything, then sending 2000 bytes will split in 1500 and 500. or if you have 1501 it will be split in 1500 and 1. Again not optimal. IP is able to fragment up to 64K. If you send something that size, then of course it will be split by IP and you will have a whole lot of optimal packets except for the last one.

The optimal size of a packet = mtu - headers of different layers used. the layers are

• TCP or UDP
• IP
• Ethernet

be warned that the mtu is not garanteed 1492. It can be lower than that it depends on the entire network.

TCP will do all that stuff for you. Because it is in TCP's nature to try and make optimal use of the mtu. Nagle is implemented as a continuous 50ms timer and a segment is only sent if that timer expires, or if the peer is expecting an ack for something that he sent himself.

Answer 2

The answer to your general "how bad is it" question depends heavily on your network topology where the communication is taking place and type of devices.

1. If you're sending the datagrams through the internet, a large number of small packets is not optimal. TCP implements Nagle's Algorithm to avoid sending small packets with 1 or 2 byte payloads, which noted in RFC 896 can cause congestion collapse.
2. If you sending the datagrams through a network with a large number of hops, the likelyhood of dropped packets will increase as each hop may not be able to handle the same number of packets in its queues.
3. If your source and destination are in the same local network segment, it won't matter as much. You'll lose some bandwidth due to the per-packet overhead.

Depending on the type of systems that are sending and receiving, a large number of small packets will increase the churn of the network stack, drivers, and application. In an embedded system, the device may spend more CPU cycles marshaling all the packets through the network stack rather than processing the data