Dropping packets with netcat using a UDP transfer?

Question

I'm working on sending large data files between two Linux computers via a 10 Gigabit Ethernet cable and netcat with a UDP transfer, but seem to be having issues.

After running several tests, I've come to the conclusion that netcat is the issue. I've tested the UDP transfer using [UDT][1], [Tsunami-UDP]2, and a Python UDT transfer as well, and all of which have not had any packet loss issues.

On the server side, we've been doing:

cat "bigfile.txt" | pv | nc -u IP PORT

then on the client side, we've been doing:

nc -u -l PORT > "outputFile.txt"

A few things that we've noticed:

1. On one of the computers, regardless of whether it's the client or server, it just "hangs". That is to say, even once the transfer is complete, Linux doesn't kill the process and move to the next line in the terminal.
2. If we run pipe view on the receiving side as well, the incoming data rate is significantly lower than what the sending side thinks it's sending.
3. Running Wireshark doesn't show any packet loss.
4. Running the system performance monitor in Linux shows that the incoming data rate (for the receiving side) is the same as the outgoing data rate from the sending side. This is in contrast to what pipe view thinks (see #2)

We're not sure where the issue is with netcat, and if there is a way around it. Any help/insights would be greatly appreciated.

Also, for what it's worth, using netcat with a TCP transfer works fine. And, I do understand that UDP isn't known for reliability, and that packet loss should be expected, but it's the protocol we must use.

Thanks

Answer 1

Tuning the Linux networking stack is a bit complicated, as there are many components to tune to figure out where data is being dropped.

If possible/feasible, I'd recommend that you start by monitoring packet drops throughout the entire network stack. Once you've done that, you can determine where exactly packets are being dropped and then adjust tuning parameters as needed. There are a lot of different files to measure with lots of different fields. I wrote a detailed blog post about monitoring and tuning each component of the Linux networking stack from top to bottom. It's a bit difficult to summarize all the information there, but take a look, I think it can help guide you.

Answer 2

This answer (How to send only one UDP packet with netcat?) says that nc sends one packet per line. Assuming that's true, this could lead to a significantly higher number of packets than your other transfer mechanisms. Presumably, as @Smeeheey suggested, you're running out of receive buffers on the receiving end.

To cause your sending end to exit, you can add -q 1 to the command line (exit 1 second after seeing end of file).

But there's no way that the the receiving end nc can know when the transfer is complete. This is why these other mechanisms are "protocols" -- they have mechanisms built into them to communicate the bounds of a file. Raw UDP has no concept of end of file.

Answer 3

It could well be that the sending instance is sending the data too fast for the receiving instance. Note that this can occur even if you see no drops on the receiving NIC (as you seem to be saying), because the loss can occur at OS level instead. Your OS could have its UDP buffers overflowing. Run this command:

watch -d "cat /proc/net/snmp | grep -w Udp" 

To see if your RcvbufErrors field is non-zero and/or growing while your file transfer is going on.