CODEWITHSUNDEEP
pythonhtmlscrapy
user1721451
user1721451

Reputation: 247

Python Scrapy - dynamic HTML, div and span content needed

So I'm new to Scrapy and am looking to do something which is proving a little too ambitious. I'm hoping somebody out there can help guide me on how to gather and parse the info I'm after from this website.

I need to obtain the following: label1 4810 (this is generated dynamically) Business name Name Address1 Address2 Address3 Address4 Postcode 0800 111111 [email protected]

Is this even possible using scrapy?

Many thanks in advance.

<div class="mbg">
  <a href="http://www.domain.com" aria-label="label1"> <span class="nw1">Label13345</span>
  </a>
  <span class="mbg-l">
    <a href="http://www.domain.com/1" title="FBS">4810</a> 
	    <img 
	      alt="4810"
	      title="4810"
	      src="http://www.domain.com/image1"></span>
</div>
<div id="bsi-c" class=" bsi-c-uk-bislr">
  <div class="bsi-cnt">
    <div class="bsi-ttl section-ttl">
      <h2>Info</h2>
      <div class="rd-sep"></div>
    </div>
    <div class="bsi-bn">Business name</div>
    <div class="bsi-cic">
      <div id="bsi-ec" class="u-flL">
        <span class="bsi-arw"><a href="javascript:;"></a></span>
        <span class="bsi-cdt"><a href="javascript:;">Contact details</a></span>
      </div>
      <div id="e8" class="u-flL bsi-ci">
        <div class="bsi-c1">
          <div>Name</div>
          <div>Address1</div>
          <div>Address2</div>
          <div>Address3</div>
          <div>Address4</div>
          <div>Postcode</div>
        </div>
        <div class="bsi-c2">
          <br></br>
          <div>
            <span class="bsi-lbl">Phone:</span>
            <span>0800 111111</span>
          </div>
          <div>
            <span class="bsi-lbl">Email:</span>
            <span>[email protected]</span>
          </div>
        </div>
      </div>
    </div>

Upvotes: 1

Views: 345

Answers (1)

alex10
alex10

Reputation: 2756

An example of parsing the already received page might look something like this:

import lxml.html

page="""<div><span> . . .</span></div> """
doc = lxml.html.document_fromstring(page)

# get label1 4810
label = doc.cssselect('.mbg .mbg-l a')[0].text_content()
# get address
addres = doc.cssselect('.u-flL .bsi-c1')[0].text_content()
# get phone
phone = doc.cssselect('.bsi-c2 .bsi-lbl')[0].text_content()
# get mail      
mail = doc.cssselect('.bsi-c2 .bsi-lbl')[1].text_content()

if a page must be retrieved from the network can make so:

import requests, lxml.html

page  = requests.get('site_.com')
doc   = lxml.html.document_fromstring(page.text)
phone = doc.cssselect('.bsi-c2 .bsi-lbl')[0].text_content()

Upvotes: 1

Related Questions