CODEWITHSUNDEEP
pythonxpathscrapyweb-crawler
Vy Nguyen
Vy Nguyen

Reputation: 73

Scrapy: How to extract content in nested div's (xpath selector)?

Please see the below html markup. How can I use the xpath selector in Scrapy to pull content from the col-sm-7 class name in div?

I want to extract this text:

Infortrend EonNAS Pro 850X 8-bay Tower NAS with 10GbE

HTML:

<div class="pricing panel panel-primary">
   <div class="panel-heading">Infortrend Products</div>
   <div class="body">
    <div class="panel-subheading"><strong>EonNAS Pro Models</strong></div>
    <div class="row">
     <div class="col-sm-7"><strong>Infortrend EonNAS Pro 850X 8-bay Tower NAS with 10GbE</strong><br />
      <small>Intel Core i3 Dual-Core 3.3GHz Processor, 8GB DDR3 RAM (Drives Not Included)</small></div>
     <div class="col-sm-3">#ENP8502MD-0030<br />
      <strong> Our Price: $2,873.00</strong></div>
     <div class="col-sm-2">
      <form action="/addcart.asp" method="get">
       <input type="hidden" name="item" value="ENP8502MD-0030 - Infortrend EonNAS Pro 850X 8-bay Tower NAS with 10GbE (Drives Not Included)">
       <input type="hidden" name="price" value="$2873.00">
       <input type="hidden" name="custID" value="">
       <input type="hidden" name="quantity" value="1">
       <button type="submit" class="btn btn-primary center-block"><i class="fa fa-shopping-cart"></i> Add to Cart</button>
      </form>
     </div>
    </div>
   </div>
  </div>

I tried to use this command but it didn't work:

response.xpath('//div[@class="pricing panel panel-primary"]/div[@class="panel-heading"]/text()/div[@class="body"]//div[@class="panel-subheading" and contains(@style,'font-weight:bold')]/text()').extract_first()

Upvotes: 5

Views: 4239

Answers (2)

Satish Prakash Garg
Satish Prakash Garg

Reputation: 2233

You can fetch text between <strong> element, something like this :

print(response.xpath('//div[@class="col-sm-7"]//text()').extract()[0].strip())

or 

print(response.xpath('//div[@class="col-sm-7"]/strong/text()').extract()[0].strip())

Both the above statements will result in :

Infortrend EonNAS Pro 850X 8-bay Tower NAS with 10GbE

You can fetch the text between all the elements inside this div with //text() including within <strong> and <small> tags inside the element, something like this :

elem_text = ' '.join([txt.strip() for txt in response.xpath('//div[@class="col-sm-7"]//text()').extract()])
print(elem_text)

This will result in :

Infortrend EonNAS Pro 850X 8-bay Tower NAS with 10GbE  Intel Core i3 Dual-Core 3.3GHz Processor, 8GB DDR3 RAM (Drives Not Included)

Upvotes: 4

frfahim
frfahim

Reputation: 525

Try this:

response.xpath('//*[@class="col-sm-7"]//strong//text()').extract()

Hope it help :)

Upvotes: 2

Related Questions