CODEWITHSUNDEEP
vmwareesxidpdk
mrbultitude
mrbultitude

Reputation: 168

"Incompatible hardware version" error when running DPDK on VMWare with VMXNET3 interface

We're trying to run DPDK example apps in a guest machine running Centos 7.5. The host is ESXi version 6.5.

I'm building dpdk on the guest machine where I'm trying to run it. I've tried both DPDK versions 18.05 and 18.08.

We have created five interfaces on esxi for connection to our guest. One management port and four data ports. We're binding theses four data ports to DPDK. The ports are all VMXNET3 interfaces. They are basically setup like the VMXNET3 interfaces in [https://doc.dpdk.org/guides/nics/vmxnet3.html], using a vswitch to connect to a physical interface. However note that we do not have any VF interfaces as shown in this document, only VMXNET3 interfaces. Unfortunately this document does not show any details on how to do the setup.

This document from vmware also shows a very similar setup. But again no details on how to setup.

Fundamentally, the roadblock we are hitting is that the VMXNET3 interfaces are failing initialization when starting the DPDK example app. Here is what we see:

[root@rg-vm ~]# ./dpdk-18.08/examples/packet_ordering/build/packet_ordering -c 0x0e0 -- -p 0xf
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
eth_vmxnet3_dev_init(): Incompatible hardware version: 0
EAL: Requested device 0000:04:00.0 cannot be used

We see this for all four interfaces that we are trying to bind to DPDK. However, strangely, sometimes after a reboot, the first two interfaces initialize correctly. But after that first attempt, all four interfaces then fail the same way.

Here are the commands we're using to setup DPDK.

modprobe uio
insmod ./dpdk-18.08/build/build/kernel/linux/igb_uio/igb_uio.ko
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 04:00.0
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 0c:00.0
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 13:00.0
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 1b:00.0

Note that we have also tried using the uio_pci_generic with the same results. We have not been able to get the vfio-pci driver to bind to the VMXNET3 interfaces.

I'm not sure it matters, but the physical interfaces on the other side of the vswitch that we're connecting to are: 17:00.0 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network Connection (rev 01)

We have also tried using a Ethernet cards based on the intel 82576 chipset (this is the chipset DPDK shows being used in their documentation), and one based on the Intel X710. We see the same error using either of these cards as we did with the i350. So I think that eliminates the ethernet hardware, which makes sense, as using the vswitch between us and the ethernet controller should make us agnostic to what it actually is.

We are running on a Dell R540. Also note that when we run Centos 7.5 with DPDK on this hardware without VMWare, everything works fine. Also if we run in VMWare, but "passthrough" the i350 interfaces to the VM (instead of using vswitch and vmxnet) everything also works fine in that case.

I've tried updating the kernel (3.10) to the latest (4.18) but still get the same error.

If I try to read the version register (VRRS) (the one that causes this error) in the vmxnet3 pci bar registers (before I bind to DPDK) using ethtool, it looks fine (0xf). I've googled around a lot but can't seem to find much help on this. It is very possible the issue is with how I'm setting things up but I can't find any info that gives details on how else to do it.

Any help would be greatly appreciated. Thanks!

Upvotes: 2

Views: 1497

Answers (1)

Abe
Abe

Reputation: 56

Try these steps:

  1. cd /etc/default
  2. vi grub

  3. Edit GRUB-CMDLINE and Add “nopku”

    GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet nopku transparent_hugepage=never log_buf_len=8M"

  4. Recompile grub: sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  5. Reboot the VM and try DPDK.

Upvotes: 4

Related Questions