CODEWITHSUNDEEP
pythonhtml
Dython
Dython

Reputation: 9

How to grab the second "span" ? (making a web scraper in python)

I'm new to python and I'm building a webscraper. I want all the instances of the second "span" on the whole webpage. My aim is to get all the names of Brands of the cars(example : Nissan) and the car model name(example: Pathfinder)

But i don't know how to grab all the car model. I've tried indexing but cannot make a loop that gives all the model names.

Below is the page's html from which i wanna grab the names.

   <h3 class="brandModelTitle">
    <span class="txtGrey3">NISSAN</span>
    <span class="txtGrey3">PATHFINDER</span>

    <span class="version txtGrey7C noBold">(2)
    2.5 DCI 190 LE 7PL EURO5</span>

    </h3>

And below is the code i used to find all brand names Names = []

Prices_Cars = []
for var1 in soup.find_all('h3', class_ = 'brandModelTitle'):
    brand_Names = var1.span.text
    Names.append(brand_Names)

Upvotes: 1

Views: 504

Answers (2)

Felix Martinez
Felix Martinez

Reputation: 512

you can use scrapy, i only include parse function part:

def parse(self, response):
    #Remove XML namespaces
    response.selector.remove_namespaces()

    #Extract article information
    brands = response.xpath('//h3/span[1]/text()').extract()
    models = response.xpath('//h3/span[2]/text()').extract()
    details = response.xpath('//h3/span[3]/text()').extract()


    for item in zip(brands,models,details):
        scraped_info = {
            'brand' : item[0],
            'model' : item[1],
            'details' : item[2]
        }

        yield scraped_info

scrapy info: https://www.analyticsvidhya.com/blog/2017/07/web-scraping-in-python-using-scrapy/ xpath examples: https://www.w3schools.com/xml/xpath_examples.asp

Upvotes: 1

Claroja
Claroja

Reputation: 183

soup.find_all('h3', class_ = 'brandModelTitle') just returns the h3s, you should intercept every h3s to find all spans.

try this:

from bs4 import BeautifulSoup

str = """
   <h3 class="brandModelTitle">
    <span class="txtGrey3">NISSAN</span>
    <span class="txtGrey3">PATHFINDER</span>

    <span class="version txtGrey7C noBold">(2)
    2.5 DCI 190 LE 7PL EURO5</span>

    </h3>
"""

soup = BeautifulSoup(str,'html5lib')

result = []
for var1 in soup.find_all('h3', class_ = 'brandModelTitle'):
    dic = {}
    spans = var1.find_all('span', class_ = 'txtGrey3')
    dic["Brands"]=spans[0].get_text()
    dic["model"]=spans[1].get_text()
    result.append(dic)

Upvotes: 0

Related Questions