CODEWITHSUNDEEP
linuxtcpkernel
felix001
felix001

Reputation: 16751

Performance Read() and Write() to/from Linux SKB's

Based on a standard Linux system, where there is a userland application and the kernel network stack. Ive read that moving frames from user space to kernel space (and vica-versa) can be expensive in terms of CPU cycles.

My questions are,

  1. Why? and is moving the frame in one direction (i.e from user to kernel) have a higher impact.
  2. Also, how do things differ when you move into TAP based interfaces. As the frame will still be going between user/kernel space. Do the space concerns apply, or is there some form of zero-copy in play?

Upvotes: 3

Views: 324

Answers (2)

Tony
Tony

Reputation: 6158

Context Switch

Moving frames from user space to kernel space is called context switch, which is usually caused by system call (which invoke the int 0x80 interrupt).

  • Interrupt happens, entering kernel space;
  • When interrupt happens, os will store all of the registers' value into the kernel stack of a thread: ds, es, fs, eax, cr3 etc
  • Then it jumps to IRQ handler like a function call;
  • Through some common IRQ execution path, it will choose next thread to run by some algorithm;
  • The runtime info (all the registers) is loaded from next thread;
  • Back to user space;

As we can see, we will do a lot of works when moving frame into/out kernel, which is much more work than a simple function call (just setting ebp, esp, eip). That is why this behavior is relatively time-consuming.

Virtual Devices

As a virtual network devices, writing to TAP has no differences compared with writing to a /dev/xxx.

If you write to TAP, os will be interrupted like upper description, then it will copy your arguments into kernel and block your current thread (in blocking IO). Kernel driver thread will be notified in some ways (e.g. message queue) to receive the arguments and consume it.

In Andorid, there exists some zero-copy system call, and in my demo implementations, this can be done through the address translation between the user and kernel. Because kernel and user thread not share same address space and user thread's data may be changed, we usually copy data into kernel. So if we meet the condition, we can avoid copy:

  • this system call must be blocked, i.e. data won't change;
  • translate between addresses by page tables, i.e. kernel can refer to right data;

Code

The following are codes from my demo os, which is related to this question if you are interested in detail:

Upvotes: 3

Cloud
Cloud

Reputation: 19331

Addressing questions in-line:

Why? and is moving the frame in one direction (i.e from user to kernel) have a higher impact.

Moving to/from user/kernel spaces is expensive because the OS has to:

  • Validate the pointers for the copy operation.
  • Transfer the actual data.
  • Incur the usual costs involved in transitioning between user/kernel mode.

There are some exceptions to this, such as if your driver implements a strategy such as "page flipping", which effectively remaps a chunk/page of memory so that it is accessible to a userspace application. This is "close enough" to a zero copy operation.

With respect to copy_to_user/copy_from_user performance, the performance of the two functions is apparently comparable.

Also, how do things differ when you move into TAP based interfaces. As the frame will still be going between user/kernel space. Do the space concerns apply, or is there some form of zero-copy in play?

With TUN/TAP based interfaces, the same considerations apply, unless you're utilizing some sort of DMA, page flipping, etc; logic.

Upvotes: 4

Related Questions