CODEWITHSUNDEEP
pythonxpathscrapy
Nirmal Patel
Nirmal Patel

Reputation: 61

Trying to select div tag with id not working in scrapy

I am new at scraping with scrapy. I was trying to scrape an item on Amazon and for that I tried to scrape div with id but I did not get any output 

response.xpath('//*[@id="widgetContent"]')

but it returned nothing. I have even tried more divs to select with id but I guess I am doing it the wrong way it was returning an empty list. here is the full statement:

xpath('/html/body/div[1]/div[9]/div[1]/div[3]/div/div[2]/div/div/div')

this one is also not working.

Can anyone help me? I think I am making a mistake somewhere.

I have tried with Beautiful Soup but not working....

from urllib.request import Request,urlopen
page=urlopen('https://www.amazon.in/gp/goldbox/ref=gbps_ftr_s-5_9f8e_page_1?gb_f_c2xvdC01=enforcedCategories:5122348031%252C1380375031%252C1571274031%252C1355016031%252C976389031%252C1388977031%252C4772060031%252C1953148031%252C1967851031%252C1967936031%252C1968024031%252C1953602031%252C1571271031%252C1375344031%252C976392031%252C1375393031%252C976419031%252C1388867031%252C1380441031%252C2454175031%252C2454178031%252C1983338031%252C1388921031%252C2563505031%252C2563504031%252C1968401031%252C1350384031%252C1380374031%252C976442031%252C1389335031%252C1380510031%252C1951048031%252C1350380031%252C1380442031%252C3704992031%252C1375412031%252C1634753031%252C5925789031%252C2454169031%252C1389402031%252C3677697031%252C2454172031%252C2454181031%252C1389365031%252C1984443031%252C1967862031%252C1967947031%252C1968036031%252C976460031%252C1350387031%252C2563506031%252C1375392031%252C5866078031%252C1389177031%252C3704982031%252C1389375031%252C1374515031%252C1380485031%252C1375424031%252C1983320031%252C1389432031%252C1389433031%252C976445031%252C1375427031%252C976451031%252C1983397031%252C1374594031%252C1983518031%252C1389396031%252C1983578031%252C1375458031,dealTypes:LIGHTNING_DEAL%252CBEST_DEAL,discountRanges:70-,includedAccessTypes:KINDLE_CONTENT_DEAL,sortOrder:BY_SCORE&pf_rd_p=b4500f5f-e496-4b18-ab75-623b14149f8e&pf_rd_s=slot-5&pf_rd_t=701&pf_rd_i=gb_main&pf_rd_m=A1VBAL9TL5WCBF&pf_rd_r=M7CSGXF54FR9QVVN9RSQ&nocache=1573918369538&ie=UTF8')

page_html=page.read()
page_soup=BeautifulSoup(page_html,'html.parser')
page_soup.find('div',{'id':'widgetContent'})```

Upvotes: 0

Views: 285

Answers (1)

Ikram Khan Niazi
Ikram Khan Niazi

Reputation: 805

You can do this using css selector in an easy way response.css("#widgetContent ::text"). This will also work response.xpath('//div[(@id="widgetContent")])

Upvotes: 1

Related Questions