CODEWITHSUNDEEP
pythonweb-scrapingpython-requests
blountdj
blountdj

Reputation: 639

Using Web Scrapers in a different country

I had a number of web scrapers that I'd written which were searching a country specific site such as ebay.co.uk which were manipulating the url to get specific results. To do this I'm using python 3 and the requests module.

These were working fine while I was using them in the UK, I'm now trying to run these in a different country (Spain) and they no longer work.

If I use the same URL directly in the web browser the page URL loads as before so that's so. Now I'm in a different country would I need to use a proxy to access the same URL using requests (never used a proxy before)?

Many thanks.

Edit:

Ok, this still partly works, see code:

import requests, bs4

# Tesco Outlet
URL1 = "http://www.ebay.co.uk/sch/tesco_outlet/m.html?
_nkw=&_armrs=1&_ipg=&_from="

# Sold Iphones
URL2 = "http://www.ebay.co.uk/sch/i.html?LH_Auction=1&_nkw=iphone&LH_Complete=1&LH_Sold=1&rt=nc&_trksid=p2045573.m1684"

session = requests.Session()
res = requests.get(URL2)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')

results = soup.find(class_="rcnt")
print(results)

next_page = soup.find(class_="gspr next")
print(next_page)

mainContainer = soup.find(id="mainContent")
print(mainContainer)

URL1 doesn't seem bring any results, or find the next page or main container of the page. But on URL2 all seems to work fine. Both URLsare fine in the broswer, so I'm not sure why URL1 isn't producing the expected results.

Upvotes: 0

Views: 485

Answers (1)

t.m.adam
t.m.adam

Reputation: 15376

If you 're asking how to use proxy with requests , you could make a dictionary and pass it in the proxies argument , eg : 

proxy = 'http://127.0.0.1:8000'  ## protocol :// host : port ##
res = requests.get(URL2, proxies={ 'http' : proxy })

Upvotes: 1

Related Questions