CODEWITHSUNDEEP
python-2.7web-scrapingscrapy
Omk
Omk

Reputation: 330

extract tag value in scrapy

i want to extract value of xpath tags in scrapy fpr example i have this

/html/body/div[3]/ul[1]/li[1]/div/p

q1

/html/body/div[3]/ul[1]/li[3]/div/p

ans1

/html/body/div[3]/ul[2]/li[1]/div/p

q2

/html/body/div[3]/ul[2]/li[2]/div/p

ans2 link:https://www.digikala.com/ajax/product/questions/980291

in a yield like this

 def parse(self, response):
        for quote in response.xpath('//html/body/main'):
            yield {
#question or answer 
#question pattern  li/div/p  or li[1]/div/p
#answer pattern ended with li[2 or higher number]/div/p
#related question and answer both have the same ul for example both are ul[1]
                'type': quote.xpath('i dont know this part').extract_first (),
                'QAnumber': quote.xpath('?').extract(),
                'text': quote.xpath('/html/body/div[3]/*/*/div/p/text()').extract(),


            }

how could i extract those 3 parts

Upvotes: 0

Views: 933

Answers (2)

Иван Васильев
Иван Васильев

Reputation: 142

 def parse(self, response):
     for quote in response.css('#product-questions-list > ul'):
         quest = response.css('.is-question > div.section > div > p::text').extract_first()
         answer = response.css('.is-answer > div.section > div > p::text').extract_first()
         yield {quest: answer}

Upvotes: 2

vezunchik
vezunchik

Reputation: 3717

Very hard to understand your question. Do you want to extract questions and answers? It will be something like this.

from w3lib.html import remove_tags
for qa in response.css('div#product-questions-list ul.c-faq__list'):
    question = qa.css('li.is-question div.section > p::text').get()
    answer = qa.css('li.is-answer div.section > p').get()
    answer = remove_tags(answer) if answer else None
    number = qa.css('li.is-question a::attr(data-question-id)')

Upvotes: 0

Related Questions