User-mode application that performs RDMA directly to nvme drive on Linux

Question

We've been developing HPC applications that take advantage of the infiniband infrastructure. One of our applications exchange data that is stored in an nvme cache with other nodes, and for that it uses RDMA posts. We think we can increase the throughput of these types of operations by transferring data directly from the nvme drive instead of copying to RAM first.

Is it there a way to transfer data from a file in an NVMe drive and write the incoming data to another file? And being able to do this from user code?

We are aware of NVMe over fabrics, but it appears to be a block storage solution and it does require root privilege for configuration.

Answer 1

You might consider if SPDK may help, but if I recall its mostly block level IO again. These folks here looked at RDMA file level transfer, which may help you along the path.

The short version is that its pretty hard to do what you're asking with a lot of infrastructure on top, given that an NVME drive doesn't have any concept of files, just LBAs, and when you're bypassing the OS and other stack cruft, that's the only way it communicate.

You can play some games (we have for past projects) where you create a scatter-gather list of blocks affiliated with a file and use that to request whats effectively a file directly, but again probably isn't going to suit your needs.