Writing data to csv file from table scrape

Question

I am having trouble figuring out how to write this file to csv. I am parsing data from a table, and can print it just fine, but when I try to write to a csv file i get the error "TypeError: write() argument must be str, not list". I'm not sure how to make my data points into a string.

Code:

from bs4 import BeautifulSoup
import urllib.request
import csv

html = urllib.request.urlopen("https://markets.wsj.com/").read().decode('utf8')
soup = BeautifulSoup(html, 'html.parser')  # parse your html

filename = "products.csv"
f = open(filename, "w")

t = soup.find('table', {'summary': 'Major Stock Indexes'})  # finds tag table with attribute summary equals to 'Major Stock Indexes'
tr = t.find_all('tr')  # get all table rows from selected table
row_lis = [i.find_all('td') if i.find_all('td') else i.find_all('th') for i in tr if i.text.strip()]  # construct list of data


f.write([','.join(x.text.strip() for x in i) for i in row_lis])

Any suggestions?

Answer 1

w.write() takes only a string as an argument, but your passing it a list of lists of strings.

csv.writerows() will write lists to a csv file.

Change your file handle f to be :

f = csv.writer(open(filename,'wb'))

and use it by replacing the last line with:

f.writerows([[x.text.strip() for x in i] for i in row_lis])

will produce a csv with contents: