Cannot use testpmd to send pkts to VM in qemu process though virtio

Question

I'm trying to test vhost-user/virtio-net. I used testpmd to send pkts (in txonly mode) to qemu VM. But all pkts were droped showed by testpmd. here is my environment:

DPDK version: 19.08
(HOST) Hugepagesize=1GB Hugepages=16

# testpmd cmd
testpmd -l 0-3 -n 4 --socket-mem 1024 --vdev 'net_vhost0,iface=/tmp/sock0,queues=1' -- -i

# qemu cmd
qemu-system-x86_64 /opt/vm/centos/vm.img \
        -cpu qemu64,+ssse3,+sse4.1,+sse4.2 \
        --enable-kvm \
        --nographic -vnc :0 \
        -smp 4 \
        -m 4096 -mem-path /dev/hugepages,share=on -mem-prealloc \
        -chardev socket,id=chr0,path=/tmp/sock0 \
        -netdev vhost-user,id=net0,chardev=chr0,queues=1,vhostforce \
        -device virtio-net-pci,netdev=net0,ioeventfd=on,mac=52:54:00:00:00:14 \
        -netdev tap,id=tapnet0,ifname=tap1,script=no,downscript=no \
        -device e1000,netdev=tapnet0

the link was established and virt-queue was initialized:

VHOST_CONFIG: new vhost user connection is 31
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0x7
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:32
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:33
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0

Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1

Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x7820ffc3
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region 0, size: 0x40000000
     guest physical addr: 0x100000000
     guest virtual  addr: 0x7f9f7fe00000
     host  virtual  addr: 0x7f83d8000000
     mmap addr : 0x7f8318000000
     mmap size : 0x100000000
     mmap align: 0x1000
     mmap off  : 0xc0000000
VHOST_CONFIG: guest memory region 1, size: 0xa0000
     guest physical addr: 0x0
     guest virtual  addr: 0x7f9ebfe00000
     host  virtual  addr: 0x7f8421788000
     mmap addr : 0x7f8421788000
     mmap size : 0xa0000
     mmap align: 0x1000
     mmap off  : 0x0
VHOST_CONFIG: guest memory region 2, size: 0xbff40000
     guest physical addr: 0xc0000
     guest virtual  addr: 0x7f9ebfec0000
     host  virtual  addr: 0x7f82580c0000
     mmap addr : 0x7f8258000000
     mmap size : 0xc0000000
     mmap align: 0x1000
     mmap off  : 0xc0000
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:37
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:38
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:32
VHOST_CONFIG: virtio is now ready for processing.

Port 0: link state change event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:39

set nic promisc in vm:

# ifconfig eth0 promisc up

testpmd show link status is up:

testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 56:48:4F:53:54:00
Device name: net_vhost0
Driver name: net_vhost
Devargs: iface=/tmp/sock0,queues=1
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: disabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload: 
  strip off 
  filter off 
  qinq(extend) off 
No RSS offload flow type is supported.
Minimum size of RX buffer: 0
Maximum configurable length of RX packet: 4294967295
Current number of RX queues: 1
Max possible RX queues: 1
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 65535
Max segment number per MTU/TSO: 65535

testpmd start send pkts:

testpmd> set fwd txonly
Set txonly packet forwarding mode
testpmd> start

stop testpmd:

Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 17056768      TX-total: 17056768
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 17056768      TX-total: 17056768
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

all pkts were droped.

Did I missed sth?

Answer 1

Looks like it has either DPDK or NUMA backed page issue. The same is working with DPDK version 19.11 LTS and 20.11 LTS.

• DPDK application: rm /tmp/sock0; sudo ./build/l2fwd --legacy-mem -l 1-2 --no-pci --vdev=net_vhost0,iface=/tmp/sock0 --vdev=net_tap0 -m 1024 -- -p 3 -T 1 --no-mac-updating

• QEMU: taskset -c 4-9 qemu-system-x86_64 -cpu host -enable-kvm -m 1024 -smp 4,sockets=1,cores=4,threads=1 \ -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on \ -numa node,memdev=mem,nodeid=0 -mem-prealloc \ -name test \ -no-reboot \ -vnc none \ -nographic \ -net user,hostfwd=tcp::10023-:22 -net nic \ -chardev socket,id=charnet0,path=/tmp/sock0 \ -netdev type=vhost-user,chardev=charnet0,queues=1,id=hostnet0 \ -device virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=fa:16:3e:52:30:73 \ -hda [disk name]

Once the VM is booted, you can login via ssh port 10023.