CODEWITHSUNDEEP
pythonregexbeautifulsoupweb-crawler
ching-yu
ching-yu

Reputation: 87

web crawler - How to remove child node in div?

I have a problem with writing a web crawler.

here is the HTML:

i don't want this
<div id="main-content">
  <div id="1">i don't want this</div>
  <div id="2">i don't want this</div>
  <div id="3">i don't want this</div>
  i want this!!!
  <span class="c1">i don't want this</span>
  <span class="c1">i don't want this</span>
</div>
i don't want this

And i wrote some python code:

import requests, json
from bs4 import BeautifulSoup
import re

res = requests.get(url)
soup =  BeautifulSoup(res.text,"html.parser")
main_content = soup.find(id="main-content")

#### problem here ####
m = [s.extract() for s in main_content('div')]
m = [s.extract() for s in main_content('span')]

# some regex for dealing string.
filtered = [ v for v in main_content.stripped_strings if v[0] not in [u'※',u'◆'] and v[:2] not in [u'--']]
content = ' '.join(filtered)
content = content.replace("-- "+url,"")
content = re.sub("[,.!?:，。！？：]"," ",content)
content = re.sub(r'(\s)+', ' ', content)
print(content)

It works sometimes, but sometimes an error occurs.

Upvotes: 0

Views: 119

Answers (1)

Dmitriy Fialkovskiy
Dmitriy Fialkovskiy

Reputation: 3235

Maybe such alternative approach would work:

from bs4 import BeautifulSoup as bs 

html = """i don't want this
<div id="main-content">
  <div id="1">i don't want this</div>
  <div id="2">i don't want this</div>
  <div id="3">i don't want this</div>
  i want this!!!
  <span class="c1">i don't want this</span>
  <span class="c1">i don't want this</span>
</div>
i don't want this"""

def get_text_without_children(tag):
    return ''.join(tag.find_all(text=True, recursive=False)).strip()

soup = bs(html, 'html.parser')
divs = soup.find_all('div', {"id" : "main-content"})
for i in divs:
    print(get_text_without_children(i))

Result:

i want this!!!

Upvotes: 1

Related Questions