Missing a loop to write a file

Question

import urllib.request
import re
import csv
import pandas as pd
from bs4 import BeautifulSoup

stocklist = ['aapl','goog','fb','amzn','COP']
for stocklist in stocklist:
    optionsUrl = urllib.request.urlopen('http://finance.yahoo.com/q?s='+stocklist).read()
    soup = BeautifulSoup(optionsUrl)
    stocksymbol = ['Symbol:',''+stocklist+'']
    optionsTable = [stocksymbol]+[
        [x.text for x in y.parent.contents]
        for y in soup.findAll('td', attrs={'class': 'yfnc_tabledata1','rtq_table': ''})
    ]
    print(optionsTable)
    my_df = pd.DataFrame(optionsTable).T
    my_df.to_csv('test.csv', index=False, header=False)

I have this piece of code. Some one suggested me to use pandas. I was able to write the data from a list into the CSV file. But the CSV file is having data only for COP but not for other stocks(the csv file has only one row of data, I am assuming it is overwriting). Can some one please tell me what i am missing or fix this code? the print(optionsTable) prints 4 rows though..

Here is the output:

[['Symbol:', 'aapl'], ['Prev Close:', '99.65'], ['Open:', '98.51'], ['Bid:', '98.95 x 1700'], ['Ask:', '98.96 x 1200'], ['1y Target Est:', '124.90'], ['Beta:', '1.48679'], ['Earnings Date:', 'Jul 19 - Jul 25 (Est.)'], ["Day's Range:", '98.48 - 99.35'], ['52wk Range:', '89.47 - 132.97'], ['Volume:', '28,454,663'], ['Avg Vol (3m):', '38,261,900'], ['Market Cap:', '541.57B'], ['P/E (ttm):', '11.01'], ['EPS (ttm):', '8.98'], ['Div & Yield:', '2.28 (2.30%) '], ['Forward P/E (1 yr):', '10.86'], ['P/S (ttm):', '2.40'], ['Ex-Dividend Date:', '05-May-16'], ['Annual EPS Est\n                      (Sep-16)\n                    :', '8.28'], ['Quarterly EPS Est\n                      (Jun-16)\n                    :', '1.39'], ['Mean Recommendation*:', '1.8'], ['PEG Ratio (5 yr expected):', '1.30']]
[['Symbol:', 'goog'], ['Prev Close:', '728.58'], ['Open:', '719.47'], ['Bid:', '717.60 x 400'], ['Ask:', '717.96 x 100'], ['1y Target Est:', '924.83'], ['Beta:', '1.032'], ['Next Earnings Date:', 'N/A'], ["Day's Range:", '716.43 - 725.86'], ['52wk Range:', '515.18 - 789.87'], ['Volume:', '1,050,710'], ['Avg Vol (3m):', '1,781,050'], ['Market Cap:', '493.43B'], ['P/E (ttm):', '29.25'], ['EPS (ttm):', '24.58'], ['Div & Yield:', 'N/A (N/A) '], ['Forward P/E (1 yr):', 'N/A'], ['P/S (ttm):', '6.41'], ['Ex-Dividend Date:', 'N/A'], ['Annual EPS Est\n                      (Dec-16)\n                    :', 'N/A'], ['Quarterly EPS Est\n                      (Jun-16)\n                    :', 'N/A'], ['Mean Recommendation*:', '1.8'], ['PEG Ratio (5 yr expected):', 'N/A']]
[['Symbol:', 'fb'], ['Prev Close:', '118.56'], ['Open:', '117.52'], ['Bid:', '116.39 x 800'], ['Ask:', '116.40 x 500'], ['1y Target Est:', '142.87'], ['Beta:', '0.840485'], ['Earnings Date:', 'Jul 27 - Aug 1 (Est.)'], ["Day's Range:", '116.26 - 118.11'], ['52wk Range:', '72.00 - 121.08'], ['Volume:', '17,257,639'], ['Avg Vol (3m):', '25,746,700'], ['Market Cap:', '333.25B'], ['P/E (ttm):', '71.26'], ['EPS (ttm):', '1.64'], ['Div & Yield:', 'N/A (N/A) '], ['Forward P/E (1 yr):', '25.25'], ['P/S (ttm):', '17.16'], ['Ex-Dividend Date:', 'N/A'], ['Annual EPS Est\n                      (Dec-16)\n                    :', 'N/A'], ['Quarterly EPS Est\n                      (Jun-16)\n                    :', 'N/A'], ['Mean Recommendation*:', '1.7'], ['PEG Ratio (5 yr expected):', 'N/A']]
[['Symbol:', 'amzn'], ['Prev Close:', '727.65'], ['Open:', '722.35'], ['Bid:', '716.25 x 500'], ['Ask:', '716.50 x 100'], ['1y Target Est:', '800.92'], ['Beta:', '1.6465'], ['Earnings Date:', 'Jul 21 - Jul 25 (Est.)'], ["Day's Range:", '714.21 - 724.98'], ['52wk Range:', '422.64 - 731.50'], ['Volume:', '3,161,899'], ['Avg Vol (3m):', '3,948,360'], ['Market Cap:', '338.47B'], ['P/E (ttm):', '295.70'], ['EPS (ttm):', '2.43'], ['Div & Yield:', 'N/A (N/A) '], ['Forward P/E (1 yr):', '72.29'], ['P/S (ttm):', '3.03'], ['Ex-Dividend Date:', 'N/A'], ['Annual EPS Est\n                      (Dec-16)\n                    :', '5.38'], ['Quarterly EPS Est\n                      (Jun-16)\n                    :', '1.10'], ['Mean Recommendation*:', '1.8'], ['PEG Ratio (5 yr expected):', '2.43']]
[['Symbol:', 'COP'], ['Prev Close:', '46.57'], ['Open:', '45.90'], ['Bid:', '44.47 x 1300'], ['Ask:', '44.48 x 2300'], ['1y Target Est:', '51.23'], ['Beta:', '1.42252'], ['Earnings Date:', 'Jul 28 - Aug 1 (Est.)'], ["Day's Range:", '44.26 - 46.12'], ['52wk Range:', '31.05 - 64.13'], ['Volume:', '8,217,057'], ['Avg Vol (3m):', '8,947,330'], ['Market Cap:', '55.11B'], ['P/E (ttm):', 'N/A'], ['EPS (ttm):', '-4.98'], ['Div & Yield:', '1.98 (4.16%) '], ['Forward P/E (1 yr):', '143.48'], ['P/S (ttm):', '2.11'], ['Ex-Dividend Date:', '18-May-16'], ['Annual EPS Est\n                      (Dec-16)\n                    :', '-2.26'], ['Quarterly EPS Est\n                      (Jun-16)\n                    :', '-0.67'], ['Mean Recommendation*:', '2.5'], ['PEG Ratio (5 yr expected):', '0.37']]

Answer 1

I would recommend you to use pandas-datareader, which is designed for the things you are going to do.

Here is a small demo:

from datetime import datetime
import pandas_datareader.data as wb

stocklist = ['AAPL','GOOG','FB','AMZN','COP']

start = datetime(2016,6,8)
end = datetime(2016,6,11)

p = wb.DataReader(stocklist, 'yahoo',start,end)

p - is a pandas panel, with which we can do funny things:

let's see what do we have in our panel

In [388]: p.axes
Out[388]:
[Index(['Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype='object'),
 DatetimeIndex(['2016-06-08', '2016-06-09', '2016-06-10'], dtype='datetime64[ns]', name='Date', freq='D'),
 Index(['AAPL', 'AMZN', 'COP', 'FB', 'GOOG'], dtype='object')]

In [389]: p.keys()
Out[389]: Index(['Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype='object')

selecting data

In [390]: p['Adj Close']
Out[390]:
                 AAPL        AMZN        COP          FB        GOOG
Date
2016-06-08  98.940002  726.640015  47.490002  118.389999  728.280029
2016-06-09  99.650002  727.650024  46.570000  118.559998  728.580017
2016-06-10  98.830002  717.909973  44.509998  116.620003  719.409973

In [391]: p['Volume']
Out[391]:
                  AAPL       AMZN        COP          FB       GOOG
Date
2016-06-08  20812700.0  2200100.0  9596700.0  14368700.0  1582100.0
2016-06-09  26419600.0  2163100.0  5389300.0  13823400.0   985900.0
2016-06-10  31462100.0  3409500.0  8941200.0  18412700.0  1206000.0

In [394]: p[:,:,'AAPL']
Out[394]:
                 Open       High        Low      Close      Volume  Adj Close
Date
2016-06-08  99.019997  99.559998  98.680000  98.940002  20812700.0  98.940002
2016-06-09  98.500000  99.989998  98.459999  99.650002  26419600.0  99.650002
2016-06-10  98.529999  99.349998  98.480003  98.830002  31462100.0  98.830002

In [395]: p[:,'2016-06-10']
Out[395]:
            Open        High         Low       Close      Volume   Adj Close
AAPL   98.529999   99.349998   98.480003   98.830002  31462100.0   98.830002
AMZN  722.349976  724.979980  714.210022  717.909973   3409500.0  717.909973
COP    45.900002   46.119999   44.259998   44.509998   8941200.0   44.509998
FB    117.540001  118.110001  116.260002  116.620003  18412700.0  116.620003
GOOG  719.469971  725.890015  716.429993  719.409973   1206000.0  719.409973

Answer 2

You're overwriting your csv everytime you loop. You should collect all the data and write them to the csv after looping:

stocklist = ['aapl','goog','fb','amzn','COP']
columns = []
data = []
for s in stocklist:
    optionsUrl = urllib.request.urlopen('http://finance.yahoo.com/q?s='+s).read()
    soup = BeautifulSoup(optionsUrl, "html.parser")
    stocksymbol = ['Symbol:', s]
    optionsTable = [stocksymbol]+[
    [x.text for x in y.parent.contents]
    for y in soup.findAll('td', attrs={'class': 'yfnc_tabledata1','rtq_table': ''})
    ]

    if not columns:
        columns = [o[0] for o in optionsTable]
    data.append(o[1] for o in optionsTable)

# create DataFrame from data
df = pd.DataFrame(data, columns=columns)
df.to_csv('test.csv', index=False)

Answer 3

You can append to a file by doing

with open('test.csv', 'a') as f:
    my_df.to_csv(f, header=False)

this