CODEWITHSUNDEEP
python
mrblah
mrblah

Reputation: 103487

Script to connect to a web page

Looking for a python script that would simply connect to a web page (maybe some querystring parameters).

I am going to run this script as a batch job in unix.

Upvotes: 2

Views: 42137

Answers (7)

Ryan
Ryan

Reputation: 123

in python 2.7:

import urllib2
params = "key=val&key2=val2" #make sure that it's in GET request format
url = "http://www.example.com"
html = urllib2.urlopen(url+"?"+params).read()
print html

more info at https://docs.python.org/2.7/library/urllib2.html

in python 3.6:

from urllib.request import urlopen
params = "key=val&key2=val2" #make sure that it's in GET request format
url = "http://www.example.com"
html = urlopen(url+"?"+params).read()
print(html)

more info at https://docs.python.org/3.6/library/urllib.request.html

to encode params into GET format:

def myEncode(dictionary):
    result = ""
    for k in dictionary: #k is the key
        result += k+"="+dictionary[k]+"&"
    return result[:-1] #all but that last `&`

I'm pretty sure this should work in either python2 or python3...

Upvotes: 2

Timothy Grant
Timothy Grant

Reputation: 141

If you need your script to actually function as a user of the site (clicking links, etc.) then you're probably looking for the python mechanize library.

Python Mechanize

Upvotes: 3

NawaMan
NawaMan

Reputation: 932

Try this:

aResp = urllib2.urlopen("http://google.com/");
print aResp.read();

Upvotes: 3

Buggabill
Buggabill

Reputation: 13901

A simple wget called from a shell script might suffice.

Upvotes: 2

Greg Buehler
Greg Buehler

Reputation: 3894

You might want to simply use httplib from the standard library.

myConnection = httplib.HTTPConnection('http://www.example.com')

you can find the official reference here: http://docs.python.org/library/httplib.html

Upvotes: 0

Sam DeFabbia-Kane
Sam DeFabbia-Kane

Reputation: 2609

What are you trying to do? If you're just trying to fetch a web page, cURL is a pre-existing (and very common) tool that does exactly that.

Basic usage is very simple:

curl www.example.com

Upvotes: 0

Mark Biek
Mark Biek

Reputation: 150739

urllib2 will do what you want and it's pretty simple to use.

import urllib
import urllib2

params = {'param1': 'value1'}

req = urllib2.Request("http://someurl", urllib.urlencode(params))
res = urllib2.urlopen(req)

data = res.read()

It's also nice because it's easy to modify the above code to do all sorts of other things like POST requests, Basic Authentication, etc.

Upvotes: 11

Related Questions