CODEWITHSUNDEEP
pythonhtmlpandasbeautifulsoup
Matty
Matty

Reputation: 283

Python, Beautiful Soup, WebScraping, Pandas, Dataframe

Complex Beautiful Soup query

I am becoming acquainted with Beautiful Soup and with Pandas' Dataframe, but I can't seem to combine the two. 

import urllib.request
from bs4 import BeautifulSoup
import pandas as pd


connection = urllib.request.urlopen('http://www.carfolio.com/specifications/models/?man=557')
soup = BeautifulSoup(connection, "html.parser", from_encoding='utf-7')

soup.decode('utf-7','ignore')

href_tag = soup.find_all(span="detail")
for href_tag in soup.body.stripped_strings:
    print(str(href_tag.encode('utf-7')))

Ultimately, my goal is to grab each car and create a data frame with the relevant information ("details"), such as horsepower, torque, weight, etc. I just don't have an idea of how to "grab" the details. Relevant HTML Code

I have looked around, and there are examples, but most of them are not accessing the "abbr title" Thanks

Upvotes: 2

Views: 1633

Answers (2)

mmachine
mmachine

Reputation: 926

For each car you may get information from li tag:

>>>from bs4 import BeautifulSoup

>>>url="""<li class="detail"><a href="car/?car=214027" class="addstable"><span class="automobile"><span class="manufacturer" title="Manufacturer">Manexall</span>, <span class="modelyear" title="Model year">1921 <abbr title="model year">MY</abbr></span> </span></a><span class="detail"> <abbr title="front engine, rear wheel drive">FR</abbr> 1143 <abbr title="cubic centimetres">cm<sup>3</sup></abbr> 13.2 <abbr title="Pferdestärke">PS</abbr> 13 <abbr title="brake horsepower">bhp</abbr> 9.7 <abbr title="kilowatts">kW</abbr> 363 <abbr title="kilograms">kg</abbr></span></li>"""

>>>soup = BeautifulSoup(url)
>>>[data.get_text() for data in soup.select('li')]
['Manexall, 1921 MY \n FR 1143 cm3 13.2 PS 13 bhp 9.7 kW 363 kg']

Upvotes: 1

alecxe
alecxe

Reputation: 473903

If you are okay with making an additional request for each car in the list, then here is an example working demo how to grab the car characteristics:

>>> import requests
>>> from bs4 import BeautifulSoup
>>> 
>>> soup = BeautifulSoup(requests.get("http://www.carfolio.com/specifications/models/car/?car=427691").content)
>>> for item in soup.select("div.summary dl dt"):
...     print(item.get_text(strip=True), item.find_next_sibling("dd").get_text(strip=True))
... 
(u'What body style?', u'hatchback with 4/5 seats')
(u'How long?', u'3973mm')
(u'How heavy?', u'1110kg')
(u'What size engine?', u'1 litre, 999cm3')
(u'How many cylinders?', u'3, Straight')
(u'How much power?', u'95PS/ 94bhp/ 70kW@ 5000-5500rpm')
(u'How much torque?', u'160Nm/ 118ft.lb/ 16.3kgm@ 1500-3500rpm')
(u'How quick?', u'0-100km/h: 10.9s')
(u'How fast?', u'186km/h, 116mph')
(u'How economical?', u'5.0/3.7/4.2 l/100km urban/extra-urban/combined')
(u'Whatcarbon dioxide emissions?', u'97.0CO2g/km')

Upvotes: 1

Related Questions