CODEWITHSUNDEEP
python-2.7web-scrapingbeautifulsouppython-requests
Jodmoreira
Jodmoreira

Reputation: 105

How to Run Multiples URLs in Requests from a File

I'm trying to scrap multiples websites from URLs in a txt file. There's one url per line.

my code is:

Import requests
from bs4 import BeautifulSoup

file = open('url.txt', 'r')
filelines = file.readline()
urllist = requests.get(filelines)
soup = BeautifulSoup(urllist.content, "html.parser")
content = soup.find_all("span", {"class": "title-main-info"})
print content

But it prints only the last url content (last line). What i'm doing wrong? Thanks

Upvotes: 0

Views: 38

Answers (1)

SIM
SIM

Reputation: 22440

Try this. It should work:

import requests
from bs4 import BeautifulSoup

with open('url.txt', 'r') as f:
    for links in f.readlines():
        urllist= requests.get(links.strip())
        soup = BeautifulSoup(urllist.content, "html.parser")
        content = soup.find_all("span", {"class": "title-main-info"})
        print content

Upvotes: 1

Related Questions