CODEWITHSUNDEEP
pythonpython-3.xbeautifulsoupspyder
K. Sanjay
K. Sanjay

Reputation: 125

Scrape multiple pages with Beautiful soup

I am trying to scrape multiple pages of a url.

But am able to scrape only the first page is there is a way to get all the pages.

Here is my code.

from bs4 import BeautifulSoup as Soup 
import urllib, requests, re, pandas as pd

pd.set_option('max_colwidth',500)    # to remove column limit (Otherwise, we'll lose some info) 
df = pd.DataFrame()

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:
        comp_name = elem.find('span', attrs={'class':'company'}).getText().strip()
        job_title = elem.find('a', attrs={'class':'turnstileLink'}).attrs['title']
        home_url = "http://www.indeed.com"
        job_link = "%s%s" % (home_url,elem.find('a').get('href'))
        job_addr = elem.find('span', attrs={'class':'location'}).getText()
        date_posted = elem.find('span', attrs={'class': 'date'}).getText()
        description = elem.find('span', attrs={'class': 'summary'}).getText().strip()


        comp_link_overall = elem.find('span', attrs={'class':'company'}).find('a')
        if comp_link_overall != None:
        comp_link_overall = "%s%s" % (home_url, comp_link_overall.attrs['href'])
        else: comp_link_overall = None

        df = df.append({'comp_name': comp_name, 'job_title': job_title,
                'job_link': job_link, 'date_posted': date_posted,
                'overall_link': comp_link_overall, 'job_location': job_addr, 'description': description
                }, ignore_index=True)


df

df.to_csv('path\\web_scrape_Indeed.csv', sep=',', encoding='utf-8')

Please suggest if there is anyway.

Upvotes: 1

Views: 703

Answers (1)

Xero Smith
Xero Smith

Reputation: 2076

Case 1: The code presented here is exactly what you have

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

for elem in targetElements:

The problem here is targetElements changes with every iteration in the first for loop.

To avoid this, indent the second for loop inside the first like so:

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:

Case 2: Your the bug is not a result of improper indentation (i.e. not like what is in your original post) If it is the case that your code is properly idented , then it may be the case that targetElements is an empty list. This means target.findAll('div', class_ =' row result') does not return anything. In that case, visit the sites, check out the dom, then modify your scraping program.

Upvotes: 1

Related Questions