CODEWITHSUNDEEP
pythonweb-scraping
Taylor Rhodes
Taylor Rhodes

Reputation: 33

How do I webscrape by manipulating the URL? Python 3.5

I want to scrape a table of stock data from a website this table In my code, I generate an array of stock symbols. The URL for the website finviz generates tables for each particular stock with the last portion of the URL (ei. https://finviz.com/quote.ashx?t=MBOT , and MBOT). I want to input my generated array as the final input of the URL (ei. if my array is [AAPL, MBOT] then https://finviz.com/quote.ashx?t=AAPL then https://finviz.com/quote.ashx?t=MBOT) scraping the output table from each URL and inputting the scraped information into a CSV file (in this case titled 'output.csv') Here is my code:

import csv
import urllib.request
from bs4 import BeautifulSoup

twiturl = "https://twitter.com/ACInvestorBlog"
twitpage = urllib.request.urlopen(twiturl)
soup = BeautifulSoup(twitpage,"html.parser")

print(soup.title.text)

tweets = [i.text for i in soup.select('a.twitter-cashtag.pretty-link.js-nav b')]
print(tweets)

url_base = "https://finviz.com/quote.ashx?t="
url_list = [url_base + tckr for tckr in tweets]
fpage = urllib.request.urlopen(url_list)
fsoup = BeautifulSoup(fpage, 'html.parser')

with open('output.csv', 'wt') as file:
    writer = csv.writer(file)

    # write header row
    writer.writerow(map(lambda e : e.text, fsoup.find_all('td', {'class':'snapshot-td2-cp'})))

    # write body row
    writer.writerow(map(lambda e : e.text, fsoup.find_all('td', {'class':'snapshot-td2'})))

Here is my error list

"C:\Users\Taylor .DESKTOP-0SBM378\venv\helloworld\Scripts\python.exe" "C:/Users/Taylor .DESKTOP-0SBM378/PycharmProjects/helloworld/helloworld"
Antonio Costa (@ACInvestorBlog) | Twitter
Traceback (most recent call last):
['LINU', 'FOSL', 'LINU', 'PETZ', 'NETE', 'DCIX', 'DCIX', 'KDMN', 'KDMN', 'LINU', 'CNET', 'AMD', 'CNET', 'AMD', 'NETE', 'NETE', 'AAPL', 'PETZ', 'CNET', 'PETZ', 'PETZ', 'MNGA', 'KDMN', 'CNET', 'ITUS', 'CNET']
  File "C:/Users/Taylor .DESKTOP-0SBM378/PycharmProjects/helloworld/helloworld", line 17, in <module>
    fpage = urllib.request.urlopen(url_list)
  File "C:\Users\Taylor .DESKTOP-0SBM378\AppData\Local\Programs\Python\Python36-32\Lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\Taylor .DESKTOP-0SBM378\AppData\Local\Programs\Python\Python36-32\Lib\urllib\request.py", line 517, in open
    req.timeout = timeout
AttributeError: 'list' object has no attribute 'timeout'

Process finished with exit code 1

Upvotes: 0

Views: 122

Answers (2)

NewNewton
NewNewton

Reputation: 1065

You are passing a list to urllib.request.urlopen() instead of a string, that's all! So you were already really close.

To open all the different URLs, simply use a for loop.

for url in url_list:

    fpage = urllib.request.urlopen(url)
    fsoup = BeautifulSoup(fpage, 'html.parser')

    #scrape single page and add data to list

with open('output.csv', 'wt') as file:
    writer = csv.writer(file)

    #write datalist

Upvotes: 1

jrjames83
jrjames83

Reputation: 901

You're passing a list to the urlopen method. Try the below and it'll retrieve data from the first URL. 

fpage = urllib.request.urlopen(url_list[0])
fsoup = BeautifulSoup(fpage, 'html.parser')

Upvotes: 0

Related Questions