CODEWITHSUNDEEP
c++rdma
rafoo
rafoo

Reputation: 1642

libibverbs: mutlithreaded CQ events polling

I use the function below for listening to RDMA events. Is this function wait_event() thread-safe? That is, can I process CQ events in parallel in multiple threads? I am concerned specifically about the safety of using ibv_get_cq_event() in multiple threads.

ibv_wc RdmaBase::wait_event(bool ignore_errors)
{
    // This is kind of a coroutine instead of polling the events in a different thread

    ibv_wc ret{};

start:
    if(!m_polling)
    {
        void* ctxt;
        HENSURE_ERRNO(ibv_get_cq_event(m_comp_channel, &m_cq, &ctxt) == 0);
        ibv_ack_cq_events(m_cq, 1);
        HENSURE_ERRNO(ibv_req_notify_cq(m_cq, 0) == 0);
        m_polling = true;
    }

    while(true)
    {
        const int num_completions = ibv_poll_cq(m_cq, 1, &ret);
        HENSURE_ERRNO(num_completions >= 0);

        if(num_completions == 0)
        {
            m_polling = false;
            goto start;
        }
        else
        {
            if(!ignore_errors && ret.status != IBV_WC_SUCCESS)
            {
                FATAL_ERROR("Failed status %s (%d) for wr_id %d\n",
                            ibv_wc_status_str(ret.status),
                            ret.status,
                            (int) ret.wr_id);
            }

            break;
        }
    }

    return ret;
}

Upvotes: 0

Views: 161

Answers (1)

Roland
Roland

Reputation: 6583

Yes, ibv_get_cq_event() is thread safe. Internally it is just calling the underlying Linux read system call, and that will only return an event to one thread at a time. I'm not sure your code is doing exactly what you want, because calling ibv_req_notify_cq() before consuming any CQ entries will likely immediately generate another CQ event and wake up another polling thread (and that thread may generate another event, waking up another thread...).

In general all libibverbs functions are designed to be thread safe unless locking requirements are explicitly called out in the documentation (for example the mlx5dv interface provides for applications that want to manage locking themselves to maximize performance).

Upvotes: 2

Related Questions