CODEWITHSUNDEEP
pythonlinuxawksubprocess
Lotusmeristem
Lotusmeristem

Reputation: 75

subprocess.Popen() sends awk and grep lines differently than expected

On a CentOS 7.2 I have a file called cpuload, which contains the latest CPU load data in the following format:

last 30 sec:
average load: 0
cpu0 total load:     0
cpu1 total load:     0
cpu2 total load:     0
cpu3 total load:     1
cpu4 total load:     0
cpu5 total load:     0
cpu6 total load:     0
cpu7 total load:     0

last sec:
average load: 1
cpu0 total load:     5
cpu1 total load:     1
cpu2 total load:     1
cpu3 total load:     3
cpu4 total load:     2
cpu5 total load:     1
cpu6 total load:     0
cpu7 total load:     0

I want to get the number after the "average load:" of the "last sec" bit. Two cli commands give me that information when I run them as shell commands on the terminal: grep 'average load:' cpuload | sed -n 's/.*load: //p' | tail -n1 and awk 'NR > 2 && /average load:/ {print $3}' cpuload

But when I run them in subprocess.Popen() with Shell=True I only get stderr:

for:

import subprocess

cmd = ["grep", "'average load:'", "cpuload", "|", "sed", "-n", "'s/.*load: //p'", "|", "tail", "-n1"]
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
test.stderr.read()

I get:

b"Usage: grep [OPTION]... PATTERN [FILE]...\nTry 'grep --help' for more information.\n"

and for:

import subprocess
    
cmd = cmd = ["awk", "'NR > 2 && /average load:/ {print $3}'", "cpuload"]
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
test.stderr.read()

I also get:

b"awk: cmd. line:1: 'NR > 2 && /average load:/ {print $3}'\nawk: cmd. line:1: ^ invalid char ''' in expression\n"

Even though I avoided using a |

or if shell=True I get:

b"Usage: awk [POSIX or GNU style options] -f progfile [--] file ...\nUsage: awk [POSIX or GNU style options] [--] 'program' file ...\nPOSIX options:\t\tGNU long options: (standard)\n\t-f progfile\t\t--file=progfile\n\t-F fs\t\t\t--field-separator=fs\n\t-v var=val\t\t--assign=var=val\nShort options:\t\tGNU long options: (extensions)\n\t-b\t\t\t--characters-as-bytes\n\t-c\t\t\t--traditional\n\t-C\t\t\t--copyright\n\t-d[file]\t\t--dump-variables[=file]\n\t-e 'program-text'\t--source='program-text'\n\t-E file\t\t\t--exec=file\n\t-g\t\t\t--gen-pot\n\t-h\t\t\t--help\n\t-L [fatal]\t\t--lint[=fatal]\n\t-n\t\t\t--non-decimal-data\n\t-N\t\t\t--use-lc-numeric\n\t-O\t\t\t--optimize\n\t-p[file]\t\t--profile[=file]\n\t-P\t\t\t--posix\n\t-r\t\t\t--re-interval\n\t-S\t\t\t--sandbox\n\t-t\t\t\t--lint-old\n\t-V\t\t\t--version\n\nTo report bugs, see node `Bugs' in `gawk.info', which is\nsection `Reporting Problems and Bugs' in the printed version.\n\ngawk is a pattern scanning and processing language.\nBy default it reads standard input and writes standard output.\n\nExamples:\n\tgawk '{ sum += $1 }; END { print sum }' file\n\tgawk -F: '{ print $1 }' /etc/passwd\n"

What am I doing wrong?

Upvotes: 1

Views: 502

Answers (3)

Lotusmeristem
Lotusmeristem

Reputation: 75

The issue was with passing the awk parameter to subprocess with ' around them, as detailed here

Did not accept Ed Morton's comment as it did not specify what should have been done.

Upvotes: 0

balderman
balderman

Reputation: 23825

I have a file called cpuload, which contains the latest CPU load data ...I want to get the number after the "average load:" of the "last sec" bit

why not just to use simple python code in order to get the value you are looking for?

with open('cpuload') as f:
    lines = [l.strip() for l in f.readlines()]
    got_it = False
    for line in lines:
        if got_it:
            parts = line.split(':')
            result = parts[-1].strip()
            print(result)
            break
        if line == 'last sec:':
            got_it = True

output

1

Upvotes: 1

Arnaud Valmary
Arnaud Valmary

Reputation: 2327

First case with grep, sed, tail... and pipes. You need to use shell = True parameter for Popen method and a single string for the command. We need to put cotes around parameters:

import subprocess
cmd = "grep 'average load:' cpuload | sed -n 's/.*load: //p' | tail -n1"
output = ""
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)
while True:
    output += test.stdout.readline().decode("utf-8")
    if test.poll() is not None:
        break
print("output=<%s>" % (output))

Second case, without pipe: You don't need to use shell = True parameter for Popen method and a single string for the command. We don't put cotes around parameters:

import subprocess

cmd = ["/usr/bin/awk", "NR > 2 && /^average load:/ {print $3}", "cpuload"]
output = ""
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
while True:
    output += test.stdout.readline().decode("utf-8")
    if test.poll() is not None:
        break
print("output=<%s>" % (output))

Upvotes: 0

Related Questions