CODEWITHSUNDEEP
web-scrapingterminalscrapy
BirdBud
BirdBud

Reputation: 193

Scrapy Only Returns Empty Arrays

So I start my scrapy shell with

scrapy shell 'https://www.amazon.com/s?k=tomatoes&ref=nb_sb_noss_1'

And I am trying to scrape the title of the products so I enter

response.xpath('//span[@class="a-size-base-plus a-color-base a-text-normal"]').getall()

and get: []

And when I tried it in CSS with

response.css("span.a-size-base-plus a-color-base a-text-normal").getall()

I still get: []

I don't understand why it isn't finding the element even though I am copying and pasting the tags and classes from the site.

I have also tried just writing a-size-base-plus for the classes in XPath and CSS but I still get nothing

Upvotes: 0

Views: 207

Answers (1)

bas
bas

Reputation: 15722

You need to replace the spaces between the classes with dots if you want to match multiple classes on elements:

response.css("span.a-size-base-plus.a-color-base.a-text-normal").getall()

instead of:

response.css("span.a-size-base-plus a-color-base a-text-normal").getall()

Your selector with spaces says something like: Give me all a-text-normal elements inside a-color-base elements inside span elements with a class of a-size-base-plus. This is of course not what you want.

I've also had to set a user agent to get the correct results. See this answer on how to set the user agent using scrapy shell.

Upvotes: 1

Related Questions