Sending small UDP packets from the Linux Kernel to LOOPBACK

Question

Situation: My code is basically hacked into a driver of the Linux Kernel. I want to notify an application in user space about noteworthy raw events before they get fired off to the main system.

Steps for Solution: I found a nice example for a sending UDP packets from Kernel space here: http://kernelnewbies.org/Simple_UDP_Server. They use INADDR_LOOPBACK as target address which is exactly what I want.

As this is interrupt context I decided to use a work queue to send the packets (I got BUG: Scheduling while atomic without it). So my sending code is based on the kernelnewbies code wrapped into a work queue struct fired off with INIT_WORK and schedule_work on the main process. I am not declaring my own work queue.

I am not using the Netpoll API as this Question suggests it is not possible to send data from and to localhost. "You can't send to yourself"

Problem: The data sent from Kernel and received from my UDP receiver do rarely match. I have no idea why this happens.

Code for the dummy data for testing including the definition of the struct for the work queue:

static struct socket *sock_send;
static struct sockaddr_in addr_send;

static struct ksocket_workmessage {
    unsigned char *buf;
    int len;
    struct work_struct workmessage;
} workmsg;


unsigned char testmsg[] = {'T', 'e', 's', 't', 'i', 'n', 'g', 'm', 's', 'g', '\0'};
workmsg.buf = testmsg;
workmsg.len = 11;
INIT_WORK(&workmsg.workmessage, handle_workmessage);
schedule_work(&workmsg.workmessage);

Sending the actual packet is like "int ksocket_send" from the kernelnewbies example. Only difference is that my send_socket is static and that I have to get buf and len with container_of from the work queue. I am working in a completely static context. My handle_workmessage method is also static:

static void handle_workmessage(struct work_struct *work)
{
        struct msghdr msg;
        struct iovec iov;
        mm_segment_t oldfs;
        int size = 0;

        struct ksocket_workmessage *workmsg = container_of(work, struct ksocket_workmessage, workmessage);


        if (sock_send->sk==NULL)
             return;

        iov.iov_base = workmsg->buf;
        iov.iov_len = workmsg->len;

        msg.msg_flags = 0;
        msg.msg_name = &addr_send;
        msg.msg_namelen  = sizeof(struct sockaddr_in);
        msg.msg_control = NULL;
        msg.msg_controllen = 0;
        msg.msg_iov = &iov;
        msg.msg_iovlen = 1;
        msg.msg_control = NULL;

        oldfs = get_fs();
        set_fs(KERNEL_DS);
        size = sock_sendmsg(sock_send,&msg,workmsg->len);
        set_fs(oldfs);
}

Receiving end looks like this:

int main(int argc, char**argv)
{
int sockfd,n;
struct sockaddr_in servaddr;
socklen_t len;
unsigned char mesg[1000];

sockfd=socket(AF_INET,SOCK_DGRAM,0);

bzero(&servaddr,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr=htonl(INADDR_ANY);
servaddr.sin_port=htons(REC_PORT);
bind(sockfd,(struct sockaddr *)&servaddr,sizeof(servaddr));

for (;;)
{
  n = recv(sockfd,mesg,1000,0);
  printf("-------------------------------------------------------\n");
  mesg[n] = 0;
  printf("Received the following: %d bytes\n", n);
  printf("%s",mesg);
  printf("%c",mesg[0]);
  printf(",%c",mesg[1]);
  printf(",%c",mesg[2]);
  printf(",%c",mesg[3]);
  printf(",%c",mesg[4]);
  printf(",%c",mesg[5]);
  printf(",%c",mesg[6]);
  printf(",%c",mesg[7]);
  printf(",%c",mesg[8]);
  printf(",%c\n",mesg[9]);
  //printf("%c\n",mesg[0]);
  printf("-------------------------------------------------------\n");
  memset(mesg, 0, sizeof(mesg));
 }
}

The Output looks corrupted, even though that I always send the exact same message for testing purpose:

-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
����d����,�,�,�,d,�,�,�,,
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
,�,�,�,�,2,k,�,�,�
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�<����,<,�,�,�,,,,
                    ,=
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes


,,%,�,,,,,,
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
TestingmsgT,e,s,t,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
 ����Vk��1k ,�,�,�,�,V,k,�,�,1
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
TestingmsgT,e,s,t,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
,,,,,�,,�,,
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
,,
  ,�,,,,,�,<
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------

What could be the reason for this? As it works sometimes with the expected output "TestingmsgT,e,s,t,i,n,g,m,s,g", it shouldn't be a technical restriction. Packet fragmentation should also not happen as I only send 11 bytes. There is also no packet loss. Everytime I am sending the packet, it is also received.

UPDATE: IT WORKS.. but I don't know why first, thanks for the comment from alk, that I forgot the obvious. To log just before the data is sent. I did log before calling schedule_work. Now I log directly in my send method workmsg->buf before even stored to the void * pointer from iov. The data is already corupted there.

The struct ksocket_workmessage had a char *, my data was char [] and got assigned to the pointer of the struct.

What I did now is to change the data type within my struct ksocket_workmessage:

struct ksocket_workmessage {
        unsigned char buf[11];
        int len;
        struct work_struct workmessage;
} workmsg;

As I don't have a pointer anymore, I could not create my unsigned char testmsg[], so I went for assigning buf directly:

workmsg.buf[0] = 'T';
workmsg.buf[1] = 'e';
workmsg.buf[2] = 's';
workmsg.buf[3] = 't';
workmsg.buf[4] = 'i';
workmsg.buf[5] = 'n';
workmsg.buf[6] = 'g';
workmsg.buf[7] = 'm';
workmsg.buf[8] = 's';
workmsg.buf[9] = 'g';
workmsg.buf[10] = '\0';

If anyone could tell me where my initial approach failed I will gladly accept it as the correct answer.

Answer 1

Since it sometimes works and sometimes doesn't I'd suggest the problem is that you are looking at memory which has been free()d. Thus the contents are sometimes correct and sometimes they are mangled. Since your local buffer is fine this must be occurring in the kernel before it is copied to local memory.

Indeed is the unsigned char testmsg[] declared as a local variable?

Since the message isn't sent straight away the testmsg address that you pass is on the stack. If there are subsequent functional calls then they will over write the contents of the message before it is sent. Then you will sometimes see the correct message and sometimes not. Depending on the scheduling of the work.