CODEWITHSUNDEEP
pythonhtmlpython-3.xweb-scrapingbeautifulsoup
star_it8293
star_it8293

Reputation: 439

Scraping data using BeautifulSoup

I'm trying scrape the data into a dictionary from this site,

from bs4 import BeautifulSoup 
import requests 
from pprint import pprint

page = requests.get('https://webscraper.io/') 
soup = BeautifulSoup(page.text, "lxml")

info = []
for x in range(1,7):
    items = soup.findAll("div",{"class":f"info{x}"})
    info.append(items)

however, the HTML tags are not being removed.

Upvotes: 0

Views: 56

Answers (2)

chitown88
chitown88

Reputation: 28630

You need to use .text. Then to get in the way you want, would need to do a bit of string manipulation.

from bs4 import BeautifulSoup 
import requests 
from pprint import pprint

url = 'https://webscraper.io/'
page = requests.get(url) 
soup = BeautifulSoup(page.text, "lxml")


info = []
for x in range(1,7):
    item = soup.find("div",{"class":"info%s" %x}).text.strip().replace('\n',': ')
    info.append(item)

info = '\n'.join(info)
print (info)

Upvotes: 1

heidemarie
heidemarie

Reputation: 115

Something like this might work? (Replace the webscraper.io url with your actual request URL; Also, you'd still need to clean up the \n characters from the output):

from bs4 import BeautifulSoup 
import requests 
from pprint import pprint

page = requests.get('https://webscraper.io/') 
soup = BeautifulSoup(page.text, "lxml")

info = []
for x in range(1,7):
    items = soup.findAll("div",{"class":f"info{x}"})
    info += [item.text for item in items]

I.e. item.text, and concatenate the resulting array with info

Upvotes: 1

Related Questions