CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingcss-selectors
SIM
SIM

Reputation: 22440

Grabbing items using selector within python script

I've written some code in python to get company details and names from a webpage. I used css selector in my script to collect those items. However, when I run it I get "company details" and "contact" only the first portion separated by "br" tag out of a full string. How can i get the full portion other than what I've got?

Script I'm trying with:

import requests ; from lxml import html

tree = html.fromstring(requests.get("https://www.austrade.gov.au/SupplierDetails.aspx?ORGID=ORG8000000314&folderid=1736").text)
for title in tree.cssselect("div.contact-details"):
    cDetails = title.cssselect("h3:contains('Contact Details')+p")[0].text
    cContact = title.cssselect("h4:contains('Contact')+p")[0].text
    print(cDetails, cContact)

Elements within which the search results are:

<div class="contact-details block dark">
                <h3>Contact Details</h3><p>Company Name: Distance Learning Australia Pty Ltd<br>Phone: +61 2 6262 2964<br>Fax: +61 2 6169 3168<br>Email: <a href="mailto:[email protected]">[email protected]</a><br>Web: <a target="_blank" href="http://dla.edu.au">http://dla.edu.au</a></p><h4>Address</h4><p>Suite 108A, 49 Phillip Avenue<br>Watson<br>ACT<br>2602</p><h4>Contact</h4><p>Name: Christine Jarrett<br>Phone: +61 2 6262 2964<br>Fax: +61 2 6169 3168<br>Email: <a href="mailto:[email protected]">[email protected]</a></p>
            </div>

Results I'm getting:

Company Name: Distance Learning Australia Pty Ltd Name: Christine Jarrett

Results I'm after:

Company Name: Distance Learning Australia Pty Ltd
Phone: +61 2 6262 2964
Fax: +61 2 6169 3168
Email: [email protected]

Name: Christine Jarrett
Phone: +61 2 6262 2964
Fax: +61 2 6169 3168
Email: [email protected]

Btw, my intention is to do the aforesaid thing using selectors only, not xpath. Thanks in advance.

Upvotes: 1

Views: 1309

Answers (2)

Žilvinas Rudžionis
Žilvinas Rudžionis

Reputation: 2346

text returns first text node. If you want to iterate over all child nodes while grabbing text nodes use xpath like:

company_details = title.cssselect("h3:contains('Contact Details')+p")[0]
for node in company_details.xpath("child::node()"):
    print node

result:

Company Name: Distance Learning Australia Pty Ltd
<Element br at 0x7f625419eaa0>
Phone: +61 2 6262 2964
<Element br at 0x7f625419ed08>
Fax: +61 2 6169 3168
<Element br at 0x7f625419e940>
Email: 
<Element a at 0x7f625419e8e8>
<Element br at 0x7f625419eba8>
Web: 
<Element a at 0x7f6254155af8>

Upvotes: 1

Andersson
Andersson

Reputation: 52665

Simply replace text property with text_content() method as below to get required output:

cDetails = title.cssselect("h3:contains('Contact Details')+p")[0].text_content()
cContact = title.cssselect("h4:contains('Contact')+p")[0].text_content()

Upvotes: 1

Related Questions