CODEWITHSUNDEEP
rdma
rafoo
rafoo

Reputation: 1642

verbs: `ibv_req_notify_cq`, can it miss an event?

Consider this code:

cq = ibv_create_cq(...);

while(true)
{
  ibv_req_notify_cq(cq, 0); // 1.
  
  ibv_get_cq_event(...);

  // Empty the events
  while(ibv_poll_cq(...) > 0)
  {
    process_event(...);
  }

  ibv_ack_cq_events(...); // 2.
}

My question is the following: Between 2. and 1. (of the next iteration), if a CQE is generated, aren't we gonna miss it, because ibv_req_notify_cq() will empty the CQ? And so the event is forever "lost" ?

Upvotes: 0

Views: 101

Answers (1)

user3621602
user3621602

Reputation: 73

I think there are mixed concepts here.

ibv_req_notify_cq tells the CQ it should produce an event, when the new CQE arrives

ibv_get_cq_event - blocks until event arrives

ibv_poll_cq - polls for completions, effectively emptying it, when using in a loop like in this example

ibv_ack_cq_events - acknowledges the event

So the req_notify_cq is a one-time action - as a result only one event will be produced.

When the event arrives there might be n-CQEs that are polled, but since you didn't re-arm the CQ, you do not expect any other events to arrive for that CQ.

If there are CQEs generated after the event arrives, and before the ibv_poll_cq loop finishes, they will be polled with that loop, and they will not generate any other event.

If there are CQEs generated after the ibv_poll_cq loop (for instance during ibv_ack_cq_events), they will not be polled, and they will generate and event only after the CQ is armed again in next iteration. Arming the CQ when there is CQE waiting for being polled is fine and should work properly (at least for Intel RDMA nics).

Upvotes: 0

Related Questions