CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Venkateshwaran Selvaraj
Venkateshwaran Selvaraj

Reputation: 1785

Scraping and stripping off child tags python

I have following code structure from a website I wanna scrape. 

<span class="blk">Society/Project: <b>Sai Sparsh</b></span>
      <i class="blk">
         Built-up  Area: <b>1005 Sq.Ft.</b> 
             @ <i class="WebRupeesmall b mr_5 f14">Rs.</i>6109/sq.ft</i>

I am already scraping few data by the following code

   properties = soup.findAll('a', title=re.compile('Bedroom'))
for eachproperty in properties:
 print today,","+"http:/"+ eachproperty['href']+",", eachproperty.string+"," +",".join(re.findall("'([a-zA-Z0-9,\s]*)'", eachproperty['onclick']))

and my output is 

2013-09-05 ,http://Residential-Apartment-Flat-in-Velachery-Chennai South-3-Bedroom-bhk-for-Sale-spid-E10766779, 3 Bedroom, Residential Apartment in Velachery,E10766779,9952946340,,Dealer,Bala

So for the above defined HTML sturcture I am trying to strip and get the output as follows

Sai Sparsh, 1005 Sq.Ft, 6109/sq.ft

and attach it to the already generating output(mentioned above). I have been breaking my head to navigate down the tree and use REGEX for it.

Update

Here is what I tried with the code 

cname = soup.findAll('span', {'class':'blk'})
pmoney = soup.findAll('i',{'class':'blk'})
for eachproperty in cname:
 for each in pmoney:
  tey = re.sub('(\s{2,})', ' ', eachproperty.text)[17:]
 ting = re.sub('([0-9,\s]*)', ' ', each.text)
 print tey + ting

And my output is

Rams Jai Vignesh Built-up  Area: 1050 Sq.Ft. @ Rs.5524/sq.ft
Shrudhi Homes Built-up  Area: 1050 Sq.Ft. @ Rs.5524/sq.ft
Ashtalakshmi Homes Built-up  Area: 1050 Sq.Ft. @ Rs.5524/sq.ft
Raj Flats Built-up  Area: 1050 Sq.Ft. @ Rs.5524/sq.ft

But I want my output to not have 'Built-up Area:' ,' @ ', ' Rs '. So it should be just 

Rams Jai Vignesh ,1050  ,5524
 Shrudhi Homes   ,1050 , 5524

Upvotes: 1

Views: 137

Answers (1)

alecxe
alecxe

Reputation: 474201

Why don't just use text property:

import re
from bs4 import BeautifulSoup as Soup

soup = Soup("""<span class="blk">Society/Project: <b>Sai Sparsh</b></span>
      <i class="blk">
         Built-up  Area: <b>1005 Sq.Ft.</b>
             @ <i class="WebRupeesmall b mr_5 f14">Rs.</i>6109/sq.ft</i>""")

print re.sub('(\s{2,})', ' ', soup.text)

prints:

Society/Project: Sai Sparsh Built-up Area: 1005 Sq.Ft. @ Rs.6109/sq.ft

FYI, re.sub is here to prettify the string, since there are multiple spaces etc.

UPD: here's the scraper script for you:

import re
import urllib2
from bs4 import BeautifulSoup as Soup

html = urllib2.urlopen("http://www.99acres.com/property-in-velachery-chennai-south-ffid").read()
soup = Soup(html)

re_digit = re.compile('(\d+)')
for div in soup.find_all('div', {'class': 'sT_disc grey'}):
    try:
        project = div.find('span').find('b').text.strip()
    except:
        project = 'No project'
    area = re.findall(re_digit, div.find('i', {'class': 'blk'}).text.strip())

    print ", ".join([project] + area)

Upvotes: 2

Related Questions