CODEWITHSUNDEEP
pythondataframebeautifulsoupweb-crawler
SUSHMA KUMARI
SUSHMA KUMARI

Reputation: 95

Unable to crawl multiples URLs

I have a function which two crawl the webpage and look for a particular class and find a href tag inside it.

url="https://www.poynter.org/ifcn-covid-19-misinformation/page/220/"

def url_parse(site):
   hdr = {'User-Agent': 'Mozilla/5.0'}
   req = Request(site,headers=hdr)
   page = urlopen(req)
   soup = BeautifulSoup(page)
   return soup

def article_link(URL):
   try:
      soup=url_parse(URL)
      for i in soup.find_all("a", class_="button entry-content__button entry-content__button--smaller"):
        link=i['href']
   except:
      pass    
return link



data['article_source']=""
for i, rows in data.iterrows():
   rows['article_source']= article_link(rows['url'])

Issue

The function url_parse and article_link are working fine but when I use the function article_link to update the cell inside a datagram, it stops working after 1500 or 1000 URLs. I understand there could be an IP address with my laptop but I don't understand how to solve it because there is no error message.

Expectation

The function article_link parse all URL inside the data frame.

Upvotes: 1

Views: 62

Answers (1)

αԋɱҽԃ αмєяιcαη
αԋɱҽԃ αмєяιcαη

Reputation: 11525

import requests
from bs4 import BeautifulSoup
from concurrent.futures.thread import ThreadPoolExecutor

url = "https://www.poynter.org/ifcn-covid-19-misinformation/page/{}/"

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0'
}


def main(url, num):
    with requests.Session() as req:
        print(f"Extracting Page# {num}")
        r = req.get(url.format(num), headers=headers)
        soup = BeautifulSoup(r.content, 'html.parser')
        links = [item.get("href") for item in soup.findAll(
            "a", class_="button entry-content__button entry-content__button--smaller")]
        return links


with ThreadPoolExecutor(max_workers=50) as executor:
    futures = [executor.submit(main, url, num) for num in range(1, 238)]

for future in futures:
    print(future.result())

Upvotes: 2

Related Questions