CODEWITHSUNDEEP
pythonlinuxudp
CPayne
CPayne

Reputation: 526

UDP calls using socket.send() in Python are very slow

I am trying to write a simple Python script in a Linux env to test an Ethernet switch that must handle high frequency UDP messages that contain small payloads (12 bytes).

The basic design is I have a laptop (w/ Ubuntu VirtualBox) running a python script using socket.recvfrom() connected to a 100Mbps switch (Netgear FS105). This switch is connected to a desktop PC (w/ another Ubuntu VirtualBox) that sends UDP packets containing two bytes. These bytes are just a counter which allows me to see if packets were re-ordered or lost after transmission.

The transmission code looks like this:

    self.sock = socket.socket(socket.AF_INET,
                              socket.SOCK_DGRAM) # UDP

    period_s = 1 / 5000.0 # 5KHz
    while True:
        udp_count = chr ((i >> 8) & 0xFF) + chr( i & 0xFF )
        start_time_s = time.time()
        self.sock.sendto(udp_count, (self.UDP_BROADCAST_IP, self.UDP_PORT))
        remaining_idle_time_s = period_s - start_time_s
        time.sleep(remaining_idle_time_s)
        i += 1

What happens is every few seconds, remaining_idle_time_s comes back as negative because the self.sock.sendto() function took longer than 0.2ms.

Using cProfile, I can see that the average self.sock.sendto() call takes 0.14ms but sometimes takes as long as 3ms! Using getsockopt() I can see that that the socket send buffer is plenty large (212992).

What can I do to make sock.sendto() return faster. I am guessing the instances where it takes 3ms is due to the CPU deciding to do another task. Is there a way to prevent this context switch for my program?

Upvotes: 2

Views: 2099

Answers (3)

CPayne
CPayne

Reputation: 526

Okay so I ran some more tests and as alluded to above, this transmit limit is fundamentally limited by the fact I was using a Linux VirtualBox with a Windows host.

Even when I made the VirtualBox process in Windows the highest priority and made the python socket calls RT in the VirtualBox, the process would still get interrupted for > 5ms sometimes.

Running on a native Linux machine, I could set the transmit python process to real time using sudo chrt -r -p 99 my_python_process_id and the program could send at 10KHz no problem!

Upvotes: 0

Michele d'Amico
Michele d'Amico

Reputation: 23711

You would like to do a realtime task in a non realtime OS (Linux is not a realtime OS).

Realtime doesn't mean fast but do a task in within specified time constraints: i.e. prevent context switching is a way to do it because your task time become predictable.

Use C or write a kernel driver to do it can make it faster but not deterministic. There are some hard real time Linux implementation like RTLinux but run it in a virtual machine have no sense if your host is not realtime.

There is no way to do it in Linux and I'm very happy of that because a possibility of prevent context switch means that every application can block OS to do its so called important job....

Upvotes: 1

Sam Varshavchik
Sam Varshavchik

Reputation: 118310

The short answer is that sendto() will take however long it's going to take, and you cannot prevent the kernel from deciding to do a context switch in order to service a higher priority CPU interrupt. That's what an operating system does, after all. It's what makes Linux, Linux.

I suppose you can try to squeeze some more blood from this stone by optimizing the in-memory storage so that the buffer you're transmitting via sendto() never crosses a page boundary, and always falls into some convenient cache-line; and also maybe twiddle with the various tunable Linux kernel scheduling parameters, and get something out of it.

Whether all of that voodoo will achieve anything, anyone's guess is as good as anyone else's. Perhaps you could squeeze out a nano-second or too. Would all of that sweat and tears be worth it? I don't know.

It seems to me that if bleeding edge performance is so important to you, the last thing you'd want to use would be a general purpose operating system, like Linux (since your question is tagged "Linux"), and a high-level, virtual machine-based language like Python. You'll want to get as close to the metal as possible; this means C, and perhaps even a more appropriate platform for your application would be a specialized real-time operating system, which gives guaranteed performance to applications.

Upvotes: 1

Related Questions