No message of desired type in HW time stamps

Question

I work under multi platform hw timestamp application. I am little bit confused in linux timestamp behaviour. I got error 'No message of desired type' from recvmsg and try handle it like error. My debug code below. As I see expected behavior:

1. Sent time stamps the outgoing packet is looped back to the socket's error queue with the send time stamp attached.

2. Clone of sent packet can be received with recvmsg with flags |= MSG_ERRQUEUE.

3. recvmsg return outgoing packet with sock_extended_err (ee_errno==ENOMSG). ENOMSG is 'No message of desired type'.

So it look like linux should keep clone of outgoing packet in error queue for feature time calculation. Should I skip ENOMSG in my error handler code?

if (errno == EAGAIN || errno == EINTR || errno == ENOMSG)
    break;

Why it reported via error message? Probably it is not clear why ENOMSG expected or not?

I got: error description = 'No message of desired type' from recvmsg.

recvmsg(sock, &msg, recvmsg_flags|MSG_ERRQUEUE);

for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg))
{
    ...
    switch (cmsg->cmsg_level) {
    case SOL_IP:
        ...
        pkt.cmsg = (const struct cmsghdr*)cmsg;
        pkt.msg  = (const struct msghdr*)msg;
        print_sol_ip(&pkt);
    break;
    }
}
/* Network level L3.
 * Note that there is no TCP error queue;
 * MSG_ERRQUEUE flag can not be used for socket type SOCK_STREAM.
 * Thus, any errors can only be obtained as the value returned by the socket,
 * or through option SO_ERROR.
 */
static void print_sol_ip(struct cmh_packet *pkt)
{
    const int type = pkt->cmsg->cmsg_type;
    const struct cmsghdr *cmsg = pkt->cmsg;
    if (pkt->cmsg->cmsg_level != SOL_IP) {
        printf("Wrong handler.\n");
        return;
    }
    printf("SOL::IP::");
    switch (type) {
    case IP_RECVERR:
        printf("RECVERR::\n");
        struct sock_extended_err *err;
        struct sockaddr *sk_addr;
        struct sockaddr_in *sk_addrin;
        socklen_t sk_addrlen;
        err = (struct sock_extended_err *)CMSG_DATA(pkt->cmsg);
        if ((sk_addr = malloc(sizeof(struct sockaddr))) == NULL) return;

        /*
         * The original destination address of the datagram that caused the error is supplied via msg_name
         * For local errors, no address is passed (this can be checked with the cmsg_len member of the cmsghdr).
         */
        printf("pointer to the data: %p\n", pkt->cmsg->__cmsg_data);
        printf("data byte count, including header: %zd\n", pkt->cmsg->cmsg_len); /* CMSG_LEN */
        printf("originating protocol: %d\n", pkt->cmsg->cmsg_level);             /* SOL_SOCKET */
        printf("protocol-specific type: %d\n", pkt->cmsg->cmsg_type);            /* SCM_RIGHTS */

        printf("%s = %d  \n", "error number", err->ee_errno);
        printf("%s = %d  \n", "error origin", err->ee_origin); /* origin: SO_EE_ORIGIN_ICMP..LOCAL..NONE */
        printf("%s = %d  \n", "error type", err->ee_type);     /* type: ICMP_NET_UNREACH..ICMP_HOST_UNREACH */
        printf("%s = %d  \n", "error code", err->ee_code);
        printf("%s = %d  \n", "error pad", err->ee_pad);
        printf("%s = %d  \n", "error info", err->ee_info);
        printf("%s = %d  \n", "error data", err->ee_data);
        printf("error description = '%s'\n", strerror(err->ee_errno));

        sk_addr = (struct sockaddr*)pkt->msg->msg_name;
        sk_addrlen = pkt->msg->msg_namelen;
        sk_addrin = (struct sockaddr_in*)sk_addr;
        printf("%s:%d addrlen: %d\n", inet_ntoa(sk_addrin->sin_addr), ntohs(sk_addrin->sin_port), sk_addrlen);

        print_af(sk_addr->sa_family);
        break;

    case IP_PKTINFO:
        printf("PKTINFO::\n");
        struct in_pktinfo *pki;
        pki = (struct in_pktinfo *)CMSG_DATA(pkt->cmsg);
        printf("Source interface index %u local address %s destination address %s",
                pki->ipi_ifindex,
                inet_ntoa(pki->ipi_spec_dst),
                inet_ntoa(pki->ipi_addr));
        break;
    case IP_RECVOPTS:       /* Routing header and other options are already installed on the local host. */
        printf("IP_RECVOPTS::\n");
        break;

    case IP_RETOPTS:
        printf("IP_RETOPTS::\n");
        break;

    case IP_TOS:            /* the field is used to create network packet priorities.
                           There are several default values flag TOS: IPTOS_LOWDELAY,
                           in order to minimize delays for the traffic, IPTOS_THROUGHPUT,
                           to improve throughput, IPTOS_RELIABILITY, to increase
                           reliability, IPTOS_MINCOST, should be used for "optional data"
                           that can be sent at the minimum speed. */
        printf("IP_TOS::\n");
        break;

    case IP_TTL:            /* ip_default_ttl */
        printf("IP_TTL::\n");
        int ttl;
        int *pttl;
        pttl = (int *) CMSG_DATA(pkt->cmsg);
        ttl = *pttl;
        printf("ttl value = %d\n", ttl);
        break;
    case IP_HDRINCL:        /* Enabling this flag means that the user has already added the IP header to the top of their data. */
        printf("IP_HDRINCL::\n");
        break;
    case IP_MTU:
        printf("IP_MTU::\n");
        //TODO: getsockopt(sock, level, IP_MTU, IP_MTU_value_get, size);
        break;

    case IP_ROUTER_ALERT:       /* It transmits this socket all packets that are sent with the option
                                 * of IP Router Alert. This option is used only in the
                                 * type of sockets RAW.
                                 * */
        printf("IP_ROUTER_ALERT::\n");
        break;
    case IP_MULTICAST_TTL:
        printf("IP_MULTICAST_TTL::\n");
        break;
    default:
        printf("TYPE %d\n", cmsg->cmsg_type);
        break;

    }
}

Answer 1

It was clear that I got ENOMSG from loopback timestamped skb. Like described in linux/Documentation/networking/timestamping.txt. And finally I found that ip_recv_error() extract skb from sk->sk_error_queue (errqueue) and reset error sk->sk_err = 0. But it also check if exist additional skb's in sk_error_queue. My errqueue had several skb in sk_error_queue that is why ip_recv_error in the end check errqueue found that it had skb's.

skb2 = skb_peek(&sk->sk_error_queue);

it had skb2 and sk->sk_err was reset back to ENOMSG.

sk->sk_err = SKB_EXT_ERR(skb2)->ee.ee_errno;

skb2 previously came from

void skb_tstamp_tx(struct sk_buff *orig_skb,
        struct skb_shared_hwtstamps *hwtstamps)

serr = SKB_EXT_ERR(skb);
serr->ee.ee_errno = ENOMSG;

No matter from which queue you will read if errqueue contain skb (skb cb control buffer should contain error in ee_errno) recvmsg will report error. Because udp_recvmsg call __skb_recv_datagram and it check if struct sk containe error.

int error = sock_error(sk); 

If so it will report -1 with error that was found in skb. So it is critical for udp to read all messages from sk_error_queue. Because during last read sk->sk_err will reset (or getsockopt(SO_ERROR)). Or just make skip and it will read next time with some delay.