CODEWITHSUNDEEP
pythonpython-requests
Kamikaze_goldfish
Kamikaze_goldfish

Reputation: 861

Requests posts with Python

This is the first time I am trying requests.post() because I have always used requests.get(). So I'm trying to navigate to a website and search. I am using yellowpages.com, and before I get negative feedback about using the site to scrape or about an API, I just want to try it out. The problem I am running into is that it spits out some html that isn't remotely what I am looking for. I'll post my code below to show you what I am talking about.

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
url = "https://www.yellowpages.com"
search_terms = "Car Dealership"
location = "Jackson, MS"
q = {'search_terms': search_terms, 'geo_locations_terms': location}
page = requests.post(url, headers=headers, params=q)

print(page.text)

Upvotes: 0

Views: 63

Answers (1)

user9455968
user9455968

Reputation:

Your request boils down to

$ curl -X POST \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' \
  'https://www.yellowpages.com/?search_terms=Car Dealership&geo_locations_terms=Jackson, MS'

For this the server returns a 502 Bad Gateway status code.

The reason is that you use POST together wihy query parameters params. The two don't go well together. Use data instead:

requests.post(url, headers=headers, data=q)

Upvotes: 2

Related Questions