CODEWITHSUNDEEP
rdma
amkgi
amkgi

Reputation: 382

How irdma works?

After upgrading to Rocky Linux 8.6 I noticed that it was consuming more memory. After some time I figured out why this is happening. The driver of the installed Intel E810 (ice) network card is causing the irdma module in both of my servers:

lsmod |grep ice
ice                   770048  1 irdma

I tried to find more information about configuring the NIC to use RDMA, but there is very little information. All I found on configuration are the guides for RHEL8 on how to configure Soft-RoCE or Soft-iWARP.

But that's not enough, I still don't understand whether in my case it can work on its own, without my intervention, or not. I have the following:

[root@host01 ~]# rdma l s
link irdma0/1 state ACTIVE physical_state LINK_UP netdev ens3f0
link irdma1/1 state ACTIVE physical_state LINK_UP netdev ens3f1
link irdma2/1 state ACTIVE physical_state LINK_UP netdev ens2f0 <<--
link irdma3/1 state ACTIVE physical_state LINK_UP netdev ens2f1
link irdma4/1 state DOWN physical_state DISABLED netdev ens5f0
link irdma5/1 state DOWN physical_state DISABLED netdev ens5f1

[root@host02 ~]# rdma l s
link irdma0/1 state ACTIVE physical_state LINK_UP netdev ens3f0
link irdma1/1 state ACTIVE physical_state LINK_UP netdev ens3f1
link irdma2/1 state ACTIVE physical_state LINK_UP netdev ens2f0 <<--
link irdma3/1 state ACTIVE physical_state LINK_UP netdev ens2f1
link irdma4/1 state DOWN physical_state DISABLED netdev ens5f0
link irdma5/1 state DOWN physical_state DISABLED netdev ens5f1

[root@host01 ~]# ip -br a show ens2f0
ens2f0           UP             10.12.10.163/24

[root@host02 ~]# ip -br a show ens2f0
ens2f0           UP             10.12.10.147/24

Does this mean that within the same network and interface, both servers will use RDMA for data transfer? For example, if two servers transfer data over the ens2f0 interface (irdma2/1).

Or does each application on the servers have to support and be configured to use RDMA?

For example, as with libvirt, to perform a live migration using RDMA, you need to run the command:

virsh migrate --live --rdma-pin-all --migrateuri rdma://hostname domain qemu+tcp://hostname/system

Or another example, in gluster, you can mount a volume via RDMA:

mount -t glusterfs -o transport=rdma server1:/test-volume /mnt/glusterfs

UPD1: I read the readme from intel site, but it is not really relevant if using the native irdma module, which is put in a version of any redhat-like distribution from version 8.6.

Still my main question is described above, is additional configuration of the application on the servers and interfaces needed to use RDMA in my case or not? Will it work by itself and will the servers use RDMA when passing traffic between each other without additional configuration using the native irdma module? For example, if I set up a RabbitMQ cluster on my servers and the interfaces that RabbitMQ uses have RDMA support, will all packets for RabbitMQ be sent over RDMA?

Upvotes: 2

Views: 2389

Answers (1)

Konstantin Shalygin
Konstantin Shalygin

Reputation: 11

I'm also find out that kernel memory usage 8G is too much. I disabled RDMA & IB via blacklisted this modules via modprobe

blacklist irdma
blacklist mlx5_ib

After this, kernel memory usage decreased to 4GB (twice!)

Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory          4.3G     264.1M       4.0G
userspace memory             384.9M     180.0M     204.9M
free memory                   57.6G      57.6G          0
----------------------------------------------------------
                              62.3G      58.1G       4.2G

Upvotes: 1

Related Questions