Reputation: 69
I am using this script to scrape the author information from sciencedirect articles,but I am getting none when trying to print the value.
import requests
from bs4 import BeautifulSoup
from urllib import urlopen
import csv
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
with open('urls.txt') as inf:
urls = (line.strip() for line in inf)
for url in urls:
site = urlopen(url)
soup = BeautifulSoup(site, "lxml")
for item in soup.find_all("div", {"class": "AuthorGroups"}):
final = item.text,url
print final
In urls.txt I used these 2 urls (https://www.sciencedirect.com/science/article/pii/009286749290520M,https://www.sciencedirect.com/science/article/pii/0092867495903682)
Upvotes: 1
Views: 407
Reputation: 19184
if BeautifulSoup not returned expected value, see html response from the server.
Your request blocked because it need to set proper user-agent.
.....
headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0'}
for url in urls:
print url
site = requests.get(url, headers=headers).text
.....
Upvotes: 2