CODEWITHSUNDEEP
pythonscrapy
xlmaster
xlmaster

Reputation: 721

python spider scrapy cannot launch the code

I before used Selenium, but now client needs Scrapy framework to be used in his project.

I read and watched. I came to some points how to write first request spider. But I need more kind of assist.

import scrapy


class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    plate_num = "EA66LEE"
    start_urls = [
        f'https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto='
,
    ]


    def parse(self, response):
        for quote in response.xpath('div[@class="resultsstrip"]/a/p'):
            yield {
                'plate number': plate_num,
                'price': quote.xpath('div[@class="resultsstrip"]/a/p[@class="resultsstripprice"/text()]').get(),
            }

I want scrape url if plate number exists then grab the

web element price tag.

<a id="buy_EA66LEE" class="resultsstripplate plate" href="/buy.html?plate=EA66 LEE&amp;price=999" title="Buy now">EA66 LEE                              </a>
<p class="resultsstripprice">£999</p>

even from terminal I cannot get the right values from xpath located response.xpath('div/a/p/text()').get()

Upvotes: 1

Views: 43

Answers (1)

Alexander
Alexander

Reputation: 17291

You need to add the base to your xpath expressions. an xpath path should always start with / or ./ which represent an absolute or relative xpath path. In your case you can get all of the prices with.

response.xpath('//p[@class="resultsstripprice"]/text()').getall()

The above path is an absolute path that looks for all <p> tags with a class attribute of "resultsstripprice" and extracts the text contents of the tag.

This page is a good resource/reference for xpath syntax and notation.

Upvotes: 1

Related Questions