CODEWITHSUNDEEP
cmemoryoperating-systeminfinibandrdma
Fopa Léon Constantin
Fopa Léon Constantin

Reputation: 12363

How RDMA WRITE/READ behaves when the remote memory buffer is unpinned by the OS?

let two infiniband process execute on two machines. When one process terminates, all its resources (allocated file, buffers, ...) are freed even the memory that the process registered for RDMA WRITE/READ operations.

Can someone has references on how the remote Channel Adapter on the other process knows that the buffer memory is now free ? Which element of the infiniband stack does this control and when ?

Upvotes: 1

Views: 1133

Answers (2)

Anji M
Anji M

Reputation: 11

When a connected peer exits without sending a disconnect, the IB core initiates a clean up procedure which also sends a RDMA DISCONNECT MAD packet to the other RNIC if it's RC QP. On seeing a disconnect, the other end RNIC cleansup it's SQs/RQs and stuff

Upvotes: 0

haggai_e
haggai_e

Reputation: 4820

When a Linux process terminates all its open file descriptors are closed. A process that uses RDMA would have an open device file connected with the ib_uverbs module. When that file descriptor is closed the module cleans up every open RDMA resource opened by the process, including any memory region and queue pair. This clean up includes notifying the HCA's driver and through it the device itself.

If the remote machine continues attempting to perform RDMA operations, the HCA will refuse its requests, because the relevant resources would be closed.

Upvotes: 2

Related Questions