CODEWITHSUNDEEP
pythonpython-3.xweb-crawler
yoOwhatUP
yoOwhatUP

Reputation: 23

How do I build a Web Crawler using Python 3?

I have been watching 3 videos of thenewboston on how to make a webcrawler. It seems they are outdated and the links are not there. I would be thankfull if someone could fullfill the first part of the tutorial. This is as far as I have come. I have tried different websites but to no avail.

import requests
from bs4 import BeautifulSoup

def my_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = """Here I tried different websites""" + str(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text)
        for link in soup.findALL("a", {"class" : "item-name"}):
            href = """example site like following: https://example.com""" + link.get("href")
            print(href)
        page += 1

print(my_spider(2))

Using this code I would for example want to crawl to the site for title links or something similiar.

FYI here is the video. https://www.youtube.com/watch?v=sVNJOiTBi_8&list=PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_&index=26

It is tutorial 25-27

Thanks in advance!

Upvotes: 2

Views: 3137

Answers (1)

fhd
fhd

Reputation: 11

There are libraries for python 3 to web scrape but none of them are as popular as the python 2 library Scrapy ,

Some of the python 3 Web Scrapers/Crawler available: 1- Pomp 2- Pholcidae 3- pyburrow

i didn't use any myself , but you check their site or source code at github to get a better understanding of how you use them

Upvotes: 1

Related Questions