CODEWITHSUNDEEP
pythonlinuxpython-3.xsubprocess
user1377384
user1377384

Reputation: 89

Subprocess Python.. Stringing together commands

I am writing a Python program that needs to return the active hosts scanned in one of my vulnerability scans. I have used this method before returning XML, but when I try to tack on these extra programs such as cut and grep I run into issues. Perhaps it doesn't like "pipes" | or maybe I am doing something completely wrong here with my commas but I have tried all sorts of things and cant seems to get it to return the result like it does when I run the command standalone from the command line. Thanks very much for any help that is provided.

def activeHostsQuery():
    args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']

    activeHostsNumber = subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0]
    return activeHostsNumber

Upvotes: 0

Views: 2320

Answers (3)

srgerg
srgerg

Reputation: 19329

I would try this:

def activeHostsQuery():
    args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']

    activeHostsNumber = subprocess.Popen(" ".join("'%s'" % a for a in args), shell=True, stdout=subprocess.PIPE).communicate()[0]
    return activeHostsNumber

Edit: added quotes around arguments.

Another edit: Ok, try making the command a single string:

def activeHostsQuery():
    cmd = 'curl -s -k -H \'X-Requested-With: curl demoapp\' -u username:password \'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv\' | cut -d, -f1 | sort | uniq | grep -E \'"[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}"\' | wc -l'

    ctiveHostsNumber = subprocess.Popen(cmd, shell = True, stdout = subprocess.PIPE).communicate()[0]
    return activeHostsNumber

Upvotes: 0

Charles Duffy
Charles Duffy

Reputation: 295463

The right way to string together commands -- if you want to keep the shell out of it, which you should -- is to create multiple Popen objects.

def activeHostsQuery():
    args1 = ['curl', '-s', '-k',
             '-H', 'X-Requested-With: curl demoapp',
             '-u','username:password',
             'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv']
    args2 = ['cut', '-d', '-f1']
    args3 = ['sort', '-u']
    args4 = ['grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""']
    args5 = ['wc', '-l']

    p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
    p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE); p1.stdout.close()
    p3 = subprocess.Popen(args3, stdin=p2.stdout, stdout=subprocess.PIPE); p2.stdout.close()
    p4 = subprocess.Popen(args4, stdin=p3.stdout, stdout=subprocess.PIPE); p3.stdout.close()
    p5 = subprocess.Popen(args5, stdin=p4.stdout, stdout=subprocess.PIPE); p4.stdout.close()
    activeHostsNumber = p5.communicate()[0]
    return activeHostsNumber

The advantage of this is that there's no shell involved -- you can substitute arbitrary variables into your argument lists without concern that they'll be string-split, misinterpreted, cause redirections, or anything else, and the distinctions between arguments you use in generating your lists will be honored.

Now, in this particular case, I'd do the whole thing in native Python -- there's no reason even to use curl when you have native HTTP libraries -- but knowing how to build pipelines with subprocess.Popen is useful in any event.

Upvotes: 6

Mark Harviston
Mark Harviston

Reputation: 659

I know this doesn't answer the question... but that's shell script. If you want shell script pass an argument to sh -c (or bash or something)

    args = ['sh', '-c', 'curl -s -k -H X-Requested-With: curl demoapp -u','username:password https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv | cut -d -f1 | sort | uniq | grep -E "\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\"" | wc -l'


   count = int(cubprcess.check_output(args))

or use shell=True like some others suggested. This will definitely not work on Windows if you care about such things.

really you should probably do something like:

import requests
from csv
from StringIO import StringIO
import re

req=reqeusts.get(
    'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv',
    auth=('username','passoword'),
    headers={'X-Requested-With': 'curl demoapp'})

reader = csv.reader(StringIO(req.text))
count = 0
for line in reader:
    if re.match(r'.*\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.*',line[0]) is not None:
        count += 1

print count

Upvotes: 0

Related Questions