Corinne
Corinne

Reputation: 1

DPDK's `rte_mbuf_refcnt_update` not freeing mbufs quickly enough?

I am writing an application over DPDK that must handle high traffic throughput. I need shared references to the same Mbuf, and I would like to use DPDK's built-in rte_mbuf_refcnt_update for performance.

When I try to do this, the application mempool is quickly exhausted. I looked into it, and it seems like the rte_mbuf_refcnt_update doesn't release the mbuf. However, I could imagine a race condition if I did a "read-then-free"?

Here's a basic recreation of the relevant logic (Rust):

fn new(mbuf: *mut dpdk::rte_mbuf) -> Self {
   assert!(unsafe { dpdk::rte_mbuf_refcnt_read(self.raw()) } == 1);
   Self { raw: NonNull::new_unchecked(mbuf) }
}

fn ref(mbuf: *mut dpdk::rte_mbuf) -> Self { 
    unsafe { dpdk::rte_mbuf_refcnt_update(mbuf, 1); }
    Self { raw: NonNull::new_unchecked(mbuf) }
}

// ...

// impl Drop
// Option 1: just decrement refcnt
fn drop(&mut self) {
    unsafe { dpdk::rte_mbuf_refcnt_update(self.raw_mut(), -1); }
    // Would this also free the Mbuf?
}

// Alternative: if refcount == 1, free
// impl Drop
fn drop(&mut self) {
    if unsafe { dpdk::rte_mbuf_refcnt_update(self.raw_mut(), -1) } == 0 {
       unsafe { dpdk::rte_mbuf_refcnt_set(self.raw_mut(), 1) };
       // Race here - two threads invoke `rte_pktmbuf_free` -->
       // double free or invalid access? Undefined behavior if mbuf
       // re-allocated?
       unsafe { dpdk::rte_pktmbuf_free(self.raw()) };
    }
}

EDIT: Based on the responses, it looks like the following would be reasonable:

fn drop(...) {
  // This will decrement internally, but not free unless
  // no out-standing references
  unsafe { dpdk::rte_pktmbuf_free(self.raw()) };
}

Alternatively, the second alternative above in my original post should have a similar outcome. And maybe could be more efficient to avoid some of the extra logic in rte_pktmbuf_free (not sure)?

However, I am still concerned about undefined behavior in the event of multiple threads invoking free concurrently. I assume I'm missing something in the source code that protects against this.

Does this seem reasonable?

TL;DR - what's the correct way to use dpdk reference counting?

I'm using DPDK 21.08 but could upgrade.

Thank you so much!

Upvotes: 0

Views: 62

Answers (0)

Related Questions