How to test RDMA code without actual hardware?

Question

I have C++ code which makes use of infiniband verbs for RDMA communication. I need to unit test this code, and thus, the function calls related to RDMA such as ibv_get_device_list() need to succeed without any actual hardware. From my understanding, I can do the following:

1. Create my own definition of each function to return the desired value, and link to this custom definition instead of infinband/verbs.h during testing. - Turning out to be very tedious
2. Create an interface and provide real and fake implementations of each function. The real one would simply call the infiniband verbs. - Can't do this as it would require too many changes to the original code
3. Use Soft-RoCE - I need to use the same machine as both the client and server, which I haven't been able to do

Would it be possible to use gmock to mock these functions? What other options can I consider?

Answer 1

use soft iwarp and setup the system with siw.

Answer 2

If you do decide to go for option 3 (SoftRoCE), it is certainly possible to have the client and server on the same host. You can try a Vagrant box I have created to make it easy to test SoftRoCE in a VM.

Answer 3

Number 2 is the way to go. I'm going to challenge this statement:

Can't do this as it would require too many changes to the original code

If all goes well, your IDE has a "global search and replace" that can be used.

Let's fine the easiest way to abstract out your code with a minimal amount of disruptive changes:

Start by defining a class that simply wraps those C library function calls:

 class RDMA
 {
 public:
     virtual struct ibv_device **ibv_get_device_list(int *num_devices)
     {
          return ::ibv_get_device_list(num_devices);
     }

     virtual void ibv_free_device_list(struct ibv_device **list)
     {
           return ::ibv_free_device_list(list);
     }


     virtual uint64_t ibv_get_device_guid(struct ibv_device *device)
     {
          return ::ibv_get_device_guid(device);
     }
 };

Extend the above class with any other related calls you might need.

At global scope, declare an instance of the above class and a pointer to it:

 RDMA g_product_code_rdma;
 RDMA* g_ptrRMDA = &g_product_code_rdma;

Replace all your product code calls to the ibv functions to call through to the class via the global pointer. That is, change this:

 ibv_free_device_list(&list);

to be invoked as:

 g_ptrRMDA->ibv_free_device_list(&list);

Alternatively, you could declare helper functions:

ibv_device **global_ibv_get_device_list(int *num_devices)
{
     return g_ptrRDMA->ibv_get_device_list(num_devices);
}

And then replace all your calls to use the new "global" version. A simple sed\awk script or just use your IDE to globally search and replace those function calls would be the easiest approach.

At this point, your product code functions the same as before.

in your unit tests, you simply declare a MockRDMA class that inherits from the RDMA class above.

 class MockRDMA : public RDMA
 {
 public:

     ibv_device **ibv_get_device_list(int *num_devices) override
     {
          // return a fake list of devices
     }

     virtual void ibv_free_device_list(struct ibv_device **list) override
     {
          return;
     }


     virtual uint64_t ibv_get_device_guid(struct ibv_device *device) override
     {
          return 0x012345678;
     }
 };

Then you just say this at the start of your unit tests:

    MockRDMA mock;
    g_ptrRDMA = &mock;

Example:

bool test_that_thing()
{
    RDMA* original = g_ptrRDMA;
    MockRDMA mock;
    g_ptrRDMA = &mock;

    // test code to validate the code that depends on those RDMA calls

    // restore the RDMA class
    g_ptrRDMA = original;
    return result;

}