CODEWITHSUNDEEP
pythonpython-3.xpandas
ASH
ASH

Reputation: 20322

How to dynamically append multiple data frames together?

I'm loading data from a CSV into a data frame and then looping through the rows to do web queries. All my code is shown below.

import pandas as pd
from bs4 import BeautifulSoup
import requests
import pandas as pd


df = pd.read_csv('C:\\Users\\ryans\\OneDrive\\Desktop\\Briefcase\\NY Times Dates\\exchanges.csv')
print(df)

for index, row in df.iterrows():
    passin = 'https://markets.on.nytimes.com/research/markets/holidays/holidays.asp?display=market&exchange='+row["Symbol"]
    dfs = pd.read_html(passin)
    df = dfs[0]
    print(df)

My last step here is to append data frame #2 under data frame #1, and append data frame #3 under data frame #2, and so on and so on. I Googled for a solution for this, and found several techniques to append #1 under #1, and that's it.

I'm not sure how to append data frame #n to data frame #n-1. How can I do that? I'm guessing it's an increment process, but I can't get it working here.

Upvotes: 4

Views: 1131

Answers (1)

jpp
jpp

Reputation: 164693

You can use a generator expression with pd.concat:

url = 'https://markets.on.nytimes.com/research/markets/holidays/holidays.asp?display=market&exchange='

res = pd.concat(pd.read_html(f'{url}{symbol}')[0] for symbol in df['Symbol'])

If you wish to ignore index when concatenating, use the argument ignore_index=True:

gen = (pd.read_html(f'{url}{symbol}')[0] for symbol in df['Symbol'])
res = pd.concat(gen, ignore_index=True)

Upvotes: 4

Related Questions