What is the relationship between RX/TX ring and sk_buff?

Question

I know that each NIC has its RX/TX ring in RAM for OS receiving/transmitting packets. And one item(packet descriptor) in the ring includes physical address of a packet, length of a packet and etc. I wonder that does this descriptor point to a sk_buff? And what happens if the packet is a GSO packet?Is this true that one descriptor in the ring = one packet = one sk_buff?

Answer 1

I wonder that does this descriptor point to a sk_buff?

The answer is it depends on the implementation of the NIC driver. In drivers like b44 (drivers/net/ethernet/broadcom/b44.c), it uses sk_buff as DMA buffers. But in drivers like igb (drivers/net/ethernet/intel/igb/igb_main.c), it uses a continuous range of the page frames as the DMA buffers.

In b44, when package arrives, if the package is small, then a new sk_buffer is allocated and the data is copyed to that new sk_buffer. If the pageage is big, then the sk_buffer in the rx_ring is passed to the network stack, and a new sk_buffer is allocated for the rx_ring.

In igb, when package arrives, a new sk_buff is allocated, and the sk_buff will refer to the dma buffer to avoid memory copying.

Answer 2

I wonder that does this descriptor point to a sk_buff?

Not exactly. sk_buff is a software construct, roughly, a data structure containing meta information to describe some chunk of network data AND point to the data itself. So, NIC descriptor doesn't need to point to sk_buff - it may only point to a data buffer (DMA/physical address is used).

And what happens if the packet is a GSO packet?

It's quite ambiguous question to answer since such offloads may be implemented in software (say, by the network stack) and may be done in hardware.

In the former case there is nothing to discuss in terms of NIC SW descriptors - the upper layer application provides a contiguous chunk of data, and the network stack produces smaller packets from it, so that sk_buff-s handed over to the network driver already describe small packets.

In the latter case (HW offload) the network driver is supplied with huge chunks of data (by means of handing over single sk_buff-s or sk_buff chains to it), and the network driver in turn posts appropriate descriptors to NIC - it may be one descriptor pointing to a big chunk of data, or a handful of descriptors pointing to smaller parts of the same contiguous data buffer - it doesn't matter a lot since the offload magic will take place in the HW - the overall data chunk will be sliced and packet headers will be prepended accordingly yielding many smaller network packets to be put on wire.

Is this true that one descriptor in the ring = one packet = one sk_buff?

Strictly speaking, no. It depends. Your network driver may be asked to transmit one sk_buff describing one data buffer. However, your driver under certain circumstances may decide to post multiple descriptors pointing to the same chunk of data but with different offsets - i.e. submission will be done in parts and there will be multiple descriptors in the NIC's ring related to a single sk_buff. Also, one packet is not always the same as one sk_buff - a packet may be presented as a handful of segments each described with a separate sk_buff forming an sk_buff chain (please find the next and prev fields in sk_buff).

Answer 3

The Linux kernel uses an sk_buff data structure to describe each packet. When a packet arrives at the NIC, it invokes the DMA engine to place the packet into the kernel memory via empty sk_buff's stored in a ring buffer called rx_ring . An incoming packet is dropped if the ring buffer is full. When a packet is processed at higher layers, packet data remains in the same kernel memory, avoiding any extra memory copies.

http://www.ece.virginia.edu/cheetah/documents/papers/TCPlinux.pdf

That last sentence seems to indicate that incoming packet data is kept in kernel memory in sk_buff structs without redundancy. So I'd say the answer to your question is yes, that descriptor would point to an sk_buff. And yes, each packet is put in it's own sk_buff in rx_ring.

Answer 4

sk_buff has nothing to do with physical network interfaces (not directly, at least). sk_buff lists store data as seen by the socket accessing software and kernel protocol handlers (which manipulate those lists to add/remove headers and/or alter data, e.g. when encryption is employed).

It is a responsibility of a low level driver to translate sk_buff list contents into something physical network adapter will understand. In particular, network hardware can be really dumb (like when doing networking over serial lines), in which case the driver will basically read sk_buff lists byte by byte and send those over the wire.

The more advanced adapters are usually capable of doing scatter/gather DMA - given a list of addresses in RAM they will be able to access each address and either obtain a packet data from there or put a received data back. However the exact details of this mechanism are very much adapter specific and in many cases are not even consistent between single vendor's products.