CODEWITHSUNDEEP
python-3.xweb-scrapinggetpython-requests
SIM
SIM

Reputation: 22440

Facing issues working with GET request

When I run my first script it fetches results but when I run my second script, I get nothing. I do't find anything missing in my second script to get the required results. The url used in the first script is concatenated with the requirement which I found in the developer tool. Why ain't my second script working when I use the base url along with parameter?

import requests
from lxml import html
response = requests.get("http://www.ebay.com/sch/i.html?_from=R40&_trksid=p2050601.m570.l1313.TR0.TRC0.H0.Xfunny+bear.TRS0&_nkw=funny+bear&_sacat=237")
tree=html.fromstring(response.text)
titles=tree.xpath("//h3[@class='lvtitle']")
for title in titles:
    name=title.xpath(".//a[@class='vip']/text()")[0]
    print(name)

import requests
from lxml import html
payload={'_from':'R40','_trksid':'p2050601.m570.l1313.TR0.TRC0.H0.Xfunny+bear.TRS0','_nkw':'funny+bear','_sacat':'237'}
headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36'}
response = requests.get("http://www.ebay.com/",data=payload,headers=headers)
tree=html.fromstring(response.text)
titles=tree.xpath("//h3[@class='lvtitle']")
for title in titles:
    name=title.xpath(".//a[@class='vip']/text()")[0]
    print(name)

Upvotes: 0

Views: 81

Answers (1)

Ernie
Ernie

Reputation: 120

First, your URL is http://www.ebay.com/sch/i.html. (not just ebay.com)

Then, if you wanted to build a query string like wwww.example.com?key1=value1&key2=value2, you need to use params as the parameter rather than data.

This code should work.

import requests
from lxml import html
payload = {'_from':'R40','_trksid':'p2050601.m570.l1313.TR0.TRC0.H0.Xfunny+bear.TRS0','_nkw':'funny+bear','_sacat':'237'}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36'}
response = requests.get("http://www.ebay.com/sch/i.html", params=payload, headers=headers)
tree = html.fromstring(response.text)
titles = tree.xpath("//h3[@class='lvtitle']")
for title in titles:
    name = title.xpath(".//a[@class='vip']/text()")[0]
    print(name)

Reference: http://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls

Upvotes: 1

Related Questions