CODEWITHSUNDEEP
pythonpython-requests
Kamikaze_goldfish
Kamikaze_goldfish

Reputation: 861

Posting form data with requests and python

Posting form data isn't working and since my other post about this wasn't working, I figured I would try to ask the question again so maybe I can get another perspective. I am currently trying to get the requests.get(url, data=q) to work. When I print, I am getting a page not found. I have resorted just to set variables and join them to the entire URL to make it work but I really want to learn this aspect about requests. Where am I making the mistake? I am using the HTML tag attributes name=search_terms and name=geo_location_terms for the form. 

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com"
q = {'search_terms': search_terms, 'geo_locations_terms': location}
page = requests.get(url, data=q)
print(page.url)

Upvotes: 0

Views: 1230

Answers (2)

Dani Mesejo
Dani Mesejo

Reputation: 61910

Besides the issues pointed by @Lev Zakharov, you need to set the cookies in your request, like this:

import requests

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"

with requests.Session() as session:
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36',
        'Cookie': 'cookies'
    })

    q = {'search_terms': search_terms, 'geo_locations_terms': location}
    response = session.get(url, params=q)

    print(response.url)
    print(response.status_code)

Output

https://www.yellowpages.com/search?search_terms=Bars&geo_locations_terms=New+Orleans%2C+LA
200

To get the cookies you can see the requests using some Network listener for instance using Chrome Developer Tools Network tab, then replace the value 'cookies'

Upvotes: 1

Lev Zakharov
Lev Zakharov

Reputation: 2428

You have few little mistakes in your code:

  1. Check form's action parameter. Then url = "https://www.yellowpages.com/search"
  2. Second parameter is geo_location_terms not geo_locations_terms.
  3. You should pass query parameters in requests.get as params not as request data (data).

So, the final version of code:

import requests

search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search_terms, 'geo_location_terms': location}
page = requests.get(url, params=q)
print(page.url)

Result:

https://www.yellowpages.com/search?search_terms=Bars&geo_location_terms=New+Orleans%2C+LA

Upvotes: 2

Related Questions