How to get CPU usage in python 2.7 without using PSUtil

Question

Trying to get CPU usage in Python without using PSUtil.

I've tried the following but it always seems to report the same figure...

def getCPUuse():
    return(str(os.popen("top -n1 | awk '/Cpu\(s\):/ {print $2}'").readline().strip(\
)))

print(getCPUuse())

This always seems to report 3.7% even when I load up the CPU.

I have also tried the following...

str(round(float(os.popen('''grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage }' ''').readline()),2))

This always seems to return 5.12. Must admit I don't really know what the above does. If I enter grep cpu /proc/stat into the command line I get something like this...

cpu  74429 1 19596 1704779 5567 0 284 0 0 0
cpu0 19596 0 4965 422508 1640 0 279 0 0 0
cpu1 18564 1 4793 427115 1420 0 1 0 0 0
cpu2 19020 0 4861 426916 1206 0 2 0 0 0
cpu3 17249 0 4977 428240 1301 0 2 0 0 0

I'm guessing my command isn't properly extracting the values for all of my CPU cores from the above output?

My objective is to get total CPU % from my device (Raspberry PI) without using PSUtil. The figure should reflect what is displayed in the OS Task Manager.

Answer 1

What PSUtil, htop, mpstat and the like do, is reading the line starting with "cpu" (actually the first line) from /proc/stat, and then calculate a percentage from the values in that line. You can find the meaning of the values on that line in man 5 proc (search for "proc/stat").

That's also what the grep cpu /proc/stat | awk .... command you mentioned does. But the values in /proc/stat represent the times spent since last boot! Maybe they wrap around after a while, I'm not sure, but the point is that these are numbers measured over a really long time.

So if you run that command, and run it again a few seconds (, minutes or even hours) later, they won't have changed much! That's why you saw it always return 5.12.

Programs like top remember the previous values and subtract them from the newly read values. From the result a 'live' percentage can be calculated that actually reflect recent CPU load.

To do something like that in python as simply as possible, but without running external commands to read /proc/stat and do the calculations for us, we can store the values we've read into a file. The next run we can read them back in, and subtract them from the new values.

#!/usr/bin/env python2.7

import os.path

# Read first line from /proc/stat. It should start with "cpu"
# and contains times spent in various modes by all CPU's totalled.
#
with open("/proc/stat") as procfile:
    cpustats = procfile.readline().split()

# Sanity check
#
if cpustats[0] != 'cpu':
    raise ValueError("First line of /proc/stat not recognised")

#
# Refer to "man 5 proc" (search for /proc/stat) for information
# about which field means what.
#
# Here we do calculation as simple as possible:
# CPU% = 100 * time_doing_things / (time_doing_things + time_doing_nothing)
#

user_time = int(cpustats[1])    # time spent in user space
nice_time = int(cpustats[2])    # 'nice' time spent in user space
system_time = int(cpustats[3])  # time spent in kernel space

idle_time = int(cpustats[4])    # time spent idly
iowait_time = int(cpustats[5])    # time spent waiting is also doing nothing

time_doing_things = user_time + nice_time + system_time
time_doing_nothing = idle_time + iowait_time

# The times read from /proc/stat are total times, i.e. *all* times spent
# doing things and doing nothing since last boot.
#
# So to calculate  meaningful CPU % we need to know how much these values
# have *changed*.  So we store them in a file which we read next time the
# script is run.
# 
previous_values_file = "/tmp/prev.cpu"
prev_time_doing_things = 0
prev_time_doing_nothing = 0

try:
    with open(previous_values_file) as prev_file:
        prev1, prev2 = prev_file.readline().split()
    prev_time_doing_things = int(prev1)
    prev_time_doing_nothing = int(prev2)
except IOError:  # To prevent error/exception if file does not exist. We don't care.
    pass   

# Write the new values to the file to use next run
#
with open(previous_values_file, 'w') as prev_file:
    prev_file.write("{} {}\n".format(time_doing_things, time_doing_nothing))

# Calculate difference, i.e: how much the number have changed
#
diff_time_doing_things = time_doing_things - prev_time_doing_things
diff_time_doing_nothing = time_doing_nothing - prev_time_doing_nothing

# Calculate a percentage of change since last run:
#
cpu_percentage = 100.0 * diff_time_doing_things/ (diff_time_doing_things + diff_time_doing_nothing)

# Finally, output the result
#
print "CPU", cpu_percentage, "%"

Here's a version that, not unlike top, prints CPU usage every second, remembering CPU times from previous measurement in variables instead of a file:

#!/usr/bin/env python2.7

import os.path
import time

def get_cpu_times():
    # Read first line from /proc/stat. It should start with "cpu"
    # and contains times spend in various modes by all CPU's totalled.
    #
    with open("/proc/stat") as procfile:
        cpustats = procfile.readline().split()

    # Sanity check
    #
    if cpustats[0] != 'cpu':
        raise ValueError("First line of /proc/stat not recognised")

    # Refer to "man 5 proc" (search for /proc/stat) for information
    # about which field means what.
    #
    # Here we do calculation as simple as possible:
    #
    # CPU% = 100 * time-doing-things / (time_doing_things + time_doing_nothing)
    #

    user_time = int(cpustats[1])    # time spent in user space
    nice_time = int(cpustats[2])    # 'nice' time spent in user space
    system_time = int(cpustats[3])  # time spent in kernel space

    idle_time = int(cpustats[4])    # time spent idly
    iowait_time = int(cpustats[5])    # time spent waiting is also doing nothing

    time_doing_things = user_time + nice_time + system_time
    time_doing_nothing = idle_time + iowait_time

    return time_doing_things, time_doing_nothing


def cpu_percentage_loop():
    prev_time_doing_things = 0
    prev_time_doing_nothing = 0
    while True:  # loop forever printing CPU usage percentage
        time_doing_things, time_doing_nothing = get_cpu_times()
        diff_time_doing_things = time_doing_things - prev_time_doing_things
        diff_time_doing_nothing = time_doing_nothing - prev_time_doing_nothing
        cpu_percentage = 100.0 * diff_time_doing_things/ (diff_time_doing_things + diff_time_doing_nothing)

        # remember current values to subtract next iteration of the loop
        #
        prev_time_doing_things = time_doing_things
        prev_time_doing_nothing = time_doing_nothing

        # Output latest perccentage
        #
        print "CPU", cpu_percentage, "%"

        # Loop delay
        #
        time.sleep(1)

if __name__ == "__main__":
    cpu_percentage_loop()

Answer 2

That's not really easy, since most of the process you describe provide the cumulative, or total average of the CPU usage.

Maybe you can try to use the mpstat command that comes with te systat package.

So, the steps I used for the following script are:

1. Ask mpstat to generate 2 reports, one right now and the other after 1 second (mpstat 1 2)
2. Then we get the Average line (the last line)
3. The last column is the %idle column, so we get that with the $NF variable from awk
4. We use Popen from subprocess but setting shell=True to accept our pipes (|) options.
5. We execute the command (communicate())
6. Clean the output with a strip
7. And subtract that (the idle percentage) from 100, so we can get the used value.

Since it will sleep for 1 second, don't be worry that it is not an instant command.

import subprocess                                      

cmd = "mpstat 1 2 | grep Average | awk '{print $NF}'"
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)                                   
out, err = p.communicate()                   
idle = float(out.strip())                              
print(100-idle)