Get SR-IOV Virtual Function counters

Question

Suppose I've got SR-IOV passthrough enabled on the host with 2 Virtual Functions, I'm running two QEMU/KVM VM's with libvirt, each connected to a VF respectively, is there any way to see the VF counters on the host (such as rx/tx pkts)?
I've tried to use ethtool -S to see stats but I can only see the global counters of the physical function.
I found an SR-IOV counters plugin for OpenStack Ceilometer but it's a Mellanox plugin and uses a proprietary drivers on the Guest VM's.
Any help would be appreciated.

Answer 1

You can see VF counters from the host with: ip -s link show <pnic>

For example, I do ip -s link show ens1f0 and see this output corresponding with VF 22:

vf 22     link/ether 56:b8:7a:ed:41:e1 brd ff:ff:ff:ff:ff:ff, vlan 30, spoof checking off, link-state auto, trust on
    RX: bytes  packets  mcast   bcast   dropped 
         30540      206       4       3        0 
    TX: bytes  packets   dropped 
         29082      213        0 

Get these values as JSON:

ip -j -s link | jq '.[] | select(.ifname == "ens1f0") | .vfinfo_list[] | select(.vf==22)'

Answer 2

I have tested this solution on Ubuntu, running on a SRIOV enabled compute in openstack installation. Steps to get counters for individual VF of a PF. The physical NIC (25G) is running on INTEL i40 driver for XXV710 device.

#lspci |grep -i ethernet
 3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
.....
# ethtool -i eth2
driver: i40e
version: 2.7.29
firmware-version: 7.20 0x80009f02 1.2829.0
expansion-rom-version: 
bus-info: 0000:3b:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Step 1-

find for sriov dir in /sys/devices:

# find  /sys/devices/ -type d -name sriov 

/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/sriov

Step 2 : Go to this dir and do ls -lrt. You will see the VF numbers. These numbers are same as seen in output of "ip link show ' command. Do the corelation.

# find  /sys/devices/ -type d -name sriov  |xargs ls -lrt

/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/sriov:
total 0
drwxr-xr-x 3 root root 0 Jan  9 21:40 7
drwxr-xr-x 3 root root 0 Jan  9 21:40 15
drwxr-xr-x 3 root root 0 Jan  9 21:40 5
drwxr-xr-x 3 root root 0 Jan  9 21:40 13
drwxr-xr-x 3 root root 0 Jan  9 21:40 3
drwxr-xr-x 3 root root 0 Jan  9 21:40 11
drwxr-xr-x 3 root root 0 Jan  9 21:40 1
drwxr-xr-x 3 root root 0 Jan  9 21:40 8
drwxr-xr-x 3 root root 0 Jan  9 21:40 6
drwxr-xr-x 3 root root 0 Jan  9 21:40 14
drwxr-xr-x 3 root root 0 Jan  9 21:40 4
drwxr-xr-x 3 root root 0 Jan  9 21:40 12
drwxr-xr-x 3 root root 0 Jan  9 21:40 2
drwxr-xr-x 3 root root 0 Jan  9 21:40 10
drwxr-xr-x 3 root root 0 Jan  9 21:40 0
drwxr-xr-x 3 root root 0 Jan  9 21:40 9

output of ip link show

## ip link show eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 3cfdfea85df9 state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:a8:5d:f9 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 11 MAC fa:16:3e:19:8c:05, vlan 1929, spoof checking off, link-state auto, trust on
    vf 12 MAC fa:16:3e:43:88:4f, vlan 1927, spoof checking off, link-state auto, trust on
    vf 13 MAC fa:16:3e:4f:b3:d0, vlan 1925, spoof checking off, link-state auto, trust on
    vf 14 MAC fa:16:3e:56:b6:4a, vlan 1923, spoof checking off, link-state auto, trust on
    vf 15 MAC 02:f1:f1:04:01:03, vlan 1938, spoof checking off, link-state auto, trust on
you can do 1 to 1 mapping and identify which is the vf you are interested in. Assume its vf 15.

Step 3- go to dir /.

# cd 15/stats

Step 4 - do ls and cat all files , except tx_spoofed

# ls -lrt
total 0
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_errors
-r--r--r-- 1 root root 4096 Jan  9 21:40 rx_packets
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_dropped
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_spoofed
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_packets
-r--r--r-- 1 root root 4096 Jan  9 21:40 rx_bytes
-r--r--r-- 1 root root 4096 Jan  9 21:40 rx_dropped
-r--r--r-- 1 root root 4096 Jan  9 21:40 tx_bytes

# tail  *
==> rx_bytes <==
108786154191

==> rx_dropped <==
0

==> rx_packets <==
975217621

==> tx_bytes <==
433665695588

==> tx_dropped <==
0

==> tx_errors <==
0

==> tx_packets <==
590997020
# tail  *
==> rx_bytes <==
108786397727

==> rx_dropped <==
0

==> rx_packets <==
975220967

==> tx_bytes <==
433665771018

==> tx_dropped <==
0

==> tx_errors <==
0

==> tx_packets <==
590998121

These are your counters.

BR// Sanjeev Tewari

Answer 3

When you enable VFs on a host, the VFs are initially bound to a host kernel network driver module, so will appear as ethNN letting you query stats. When you then attach the VF to a guest using PCI device assignment the VF is unbound from the host kernel driver, so the ethNNN device goes away in the host. It is thus impossible to query network stats for that VF in the host.

The only way to achieve that is to not use PCI device assignment, and instead associate the VF with the guest using MACVTAP in direct mode. This is not quite as high performance as using PCI assignment, but is still pretty decent due to virtio-net design and lets you see the NIC in the host to monitor traffic.