Getting real time output from iperf3 using python's subprocess

Question

This is a follow-on to: Getting realtime output using subprocess

I'm trying to use subprocess to capture output from iperf3 in real time (using python 3.6 on windows). The goal is to leave the iperf3 session running continuously and grab the data to update a real time plot.

I created an implementation based on the referenced question (see code at end of post), but the code still waits on the first "readline" call for the iperf3 session to complete.

Output and desired behavior

My code returns the output:

Iperf test
Popen returns after: 0.012966156005859375 seconds
Readline 0 returned after: 3.2275266647338867 seconds, line was: Connecting to host 10.99.99.21, port 5201
Readline 1 returned after: 3.2275266647338867 seconds, line was: [  4] local 10.99.99.7 port 55563 connected to 10.99.99.21 port 5201
Readline 2 returned after: 3.2275266647338867 seconds, line was: [ ID] Interval           Transfer     Bandwidth
Readline 3 returned after: 3.2275266647338867 seconds, line was: [  4]   0.00-0.50   sec  27.4 MBytes   458 Mbits/sec
Readline 4 returned after: 3.2275266647338867 seconds, line was: [  4]   0.50-1.00   sec  29.0 MBytes   486 Mbits/sec
Exited

The outputs show that the first readline call doesn't return until after 3 seconds, when the iperf session completes. The desired behavior is that the readline calls 0, 1, and 2 return almost immediately, and readline call #3 returns after approx. 0.5 seconds, as soon as iperf3 has completed the first 0.5 second reporting interval.

Code

import subprocess
import time

if __name__ == "__main__":
    print('Iperf test')
    tref = time.time()

    reads_to_capture = 5
    times = [0] * reads_to_capture
    lines = [''] * reads_to_capture

    interval = 0.5
    ip = '10.99.99.21' # Iperf server IP address
    process = subprocess.Popen(f'iperf3 -c {ip} -f m -i {interval} -t 3', encoding = 'utf-8',
            stdout=subprocess.PIPE)

    print(f'Popen returns after: {time.time() - tref} seconds')

    cnt = 0

    while True:
        output = process.stdout.readline()
        if cnt < reads_to_capture: # To avoid flooding the terminal, only print the first 5
            times[cnt] = time.time() - tref
            lines[cnt] = output
            cnt = cnt + 1
        if output == '': 
            rc = process.poll()
            if rc is not None:
                break

    rc = process.poll()

    for ii in range(reads_to_capture):
        print(f'Readline {ii} returned after: {times[ii]} seconds, line was: {lines[ii].strip()}')

    print('Exited')

Answer 1

I encountered a similar problem on a Windows machine and would like to share how I managed to solve it.

I initially tried using the iperf wrapper, but it only supports Linux OSs. Then, I attempted to use a stream in Windows, but it would close as soon as the client connected. The best way I found to have iperf running and display a close to live plot of the outputs was to save the output to a text file and have Python read it every second. While this might not be the most resource-efficient or elegant solution, it was the only one that worked for me after days of searching. Here’s a step-by-step guide on how I solved it:

Step 1: Run iperf3 Server

Open a terminal and run the iperf3 server:

iperf3 -s -p 5203

Step 2: Run iperf3 Client and Append Output to a File

In another terminal tab (pay attention on where you are saving the output, and make it match the same in the python script), run the iperf3 client and append the output to a text file:

iperf3 -c 127.0.0.1 -p 5203 --forceflush >> iperf3_output.txt

Step 3: Python Script to Read the File and Plot the Data

Run the following Python code to read the output file and update the graph in real-time:

import matplotlib.pyplot as plt
import matplotlib.animation as animation
import re

# Function to parse the output and extract bandwidth values
def parse_bandwidth(line):
    match = re.search(r'(\d+\.\d+)\s+Gbits/sec', line)
    if match:
        return float(match.group(1))
    return None

# Initialize lists to store time and bandwidth values
times = []
bandwidths = []
last_line_count = 0

# Function to update the graph
def update_graph(i):
    global last_line_count
    with open('iperf3_output.txt', 'r') as f:
        lines = f.readlines()
        new_lines = lines[last_line_count:]
        if new_lines:
            for line in new_lines:
                line = line.strip()
                print(line)
                bandwidth = parse_bandwidth(line)
                if bandwidth is not None:
                    times.append(len(times))
                    bandwidths.append(bandwidth)
            last_line_count = len(lines)
            ax.clear()
            ax.plot(times, bandwidths)
            ax.set_xlabel('Time (s)')
            ax.set_ylabel('Bandwidth (Gbits/sec)')
            ax.set_title('Live Bandwidth Graph')

# Set up the plot
fig, ax = plt.subplots()
ani = animation.FuncAnimation(fig, update_graph, interval=1000)

# Show the plot
plt.show()

You can improve the graph as needed. This solution worked for me, and I hope it helps others facing similar issues.

Answer 2

In order to get the real time output from iperf3 to the subprocess.Popen you need the --forceflush flag in the iperf3 command. The --forceflush flag is introduced in iperf 3.1.5, unfortunately the official compiled iperf3.exe file have only until iperf 3.1.3.

Two solution for you,

1. get the iperf >= 3.1.5 from non official route like: https://files.budman.pw/
2. use linux's iperf3

Attached with my code:

import subprocess

my_iperf_process = subprocess.Popen(["iperf3","-c","192.168.0.1","--forceflush"],stdout=subprocess.PIPE)

for line in my_iperf_process.stdout:
    print(line)

The help message of --forceflush:

--forceflush    force flushing output at every interval

Answer 3

Sorry for my late answer. There is an API for Iperf3, luckily this comes with the standard iperf3 build/installation.

This API allows python to take the common output of iperf3.

I let you the official website of the python wrapper for iperf3. It comes with simple examples for your use. Hope I could have gave you an answer.

https://iperf3-python.readthedocs.io/en/latest/index.html