CODEWITHSUNDEEP
multithreadingrdma
Demy
Demy

Reputation: 1

RDMA: Can I safely remove spinlocks from mlx4_post_send() function for one sided RDMA operations?

I am porting linux-rdma to a platform with minimal OS support, e.g., spinlocks are not supported. My application aims to have application-threads and a worker-thread. The application-threads create and enqueue requests to a shared queue. The worker-thread continuously polls the queue and posts the requests.

Since I am accessing RDMA interface from a single-thread (the worker-thread), I am wondering if removing the spinlocks inside the mlx4_post_send(), mlx4_post_recv() and mlx4_poll_cq() is safe?

In case I would like to avoid the use of pthread_spinlocks_t while having many connections (ideally one QP per thread), what would be the best workaround? My application does not involve data races -- namely, each thread will only invoke send()/post() for its own QP.

Thanks, -dimitra

Upvotes: 0

Views: 218

Answers (1)

Greg Inozemtsev
Greg Inozemtsev

Reputation: 4671

Yes, it is safe. If you check the code for the mlx5 provider you can see that single-threaded mode was added (MLX5_SINGLE_THREADED=1). I guess nobody went back and added it to the older mlx4 provider.

The official way to handle this would be to use ibv_alloc_td and then allocate the QPs in that thread domain, but that doesn't really help in your case.

Upvotes: 0

Related Questions