CODEWITHSUNDEEP
pythoncurlpopen
Angus
Angus

Reputation: 369

Using curl in Popen in Python

I run this curl command in the unix shell and it works (see below). I was able to redirect the returned data to a file but now I want to process the data in my code instead of wasting a bunch of space in a file.

curl -k -o outputfile.txt 'obfuscatedandVeryLongAddress'
#curl command above, python representation below
addr = "obfuscatedandVeryLongAddress"
theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)

theFile.stdout is empty after this. The data returned in the curl command should be something like 4,000 lines (verified when running the command in the shell). Is the size breaking theFile.stdout? Am I doing something else wrong? I tried using:

out, err = theFile.communicate()

and then printing the out variable but still nothing

edit: formatting and clarification

Upvotes: 5

Views: 8858

Answers (3)

Eugene K
Eugene K

Reputation: 3457

You need to remove shell=True.

theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE)

Should work.

If you do shell=True, you should pass a string. Otherwise, what you're actually doing is passing those arguments -k, and addr as arguments to the shell. So if your shell is sh, what you're doing is sh 'curl' -k addr.

Upvotes: 3

Feneric
Feneric

Reputation: 861

You can put the curl command in a string like:

theFile = subprocess.Popen('curl -k {}'.format(addr), stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)

Or you can remove the shell argument:

theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE)

Or you could use the pycurl module to use libcurl library directly and skip the whole additional process.

Upvotes: 0

Brendan Abel
Brendan Abel

Reputation: 37599

Eugene's is a direct answer to your question, but I thought I'd add one on using the requests library, since it will require less code and be easier to read for anyone that needs to look at your code (and has the benefit of being cross-platform).

import requests

response = requests.get('longaddress', verify=False)
print response.text

If the response is json, you can automatically convert it to a python object

print response.json()

Upvotes: 0

Related Questions