CODEWITHSUNDEEP
pythonpython-3.xpandasdataframe
specmer
specmer

Reputation: 409

Python Pandas - How to iterate every N rows until end of dataframe

How can I iterate 100 rows at a time in a Pandas dataframe? The forex_python package seems to error out when applying the lambda function all at once.

Here is what I've tried that doesn't seem to be working:

import pandas as pd
from forex_python.converter import CurrencyRates
c = CurrencyRates()
row_processed = 0

for row in df.iterrows():
    try:
        df['USD_rate'] = df.apply(lambda x: c.get_rate(x['Currency'],'USD', x['Date']), axis = 1)
        row_processed += 1
        if row_processed == 100:
            row_processed = 0
            time.sleep(5)
    except:
        continue

Upvotes: 0

Views: 1441

Answers (2)

Jonathan Leon
Jonathan Leon

Reputation: 5648

UPDATED:

Get your currency base and dates in a list.

currencies = ['USD', 'GBP']
dates = pd.date_range(start='2021-06-01', periods=2)

# collect the dataframes
df_hold = []

for currency in currencies:
    for d in dates:
        try:
            url = 'https://theforexapi.com/api/{}/?base={}'.format(d, currency)
            r = requests.get(url).json()
            df_hold.append(pd.DataFrame(r))
        except:
            continue

# concat dataframe
pd.concat(df_hold)

           date base  rates
AUD  2021-06-01  USD  1.292
BGN  2021-06-01  USD  1.600
BRL  2021-06-01  USD  5.202
CAD  2021-06-01  USD  1.203
CHF  2021-06-01  USD  0.899
..          ...  ...    ...
SGD  2021-06-02  GBP  1.873
THB  2021-06-02  GBP 44.074
TRY  2021-06-02  GBP 12.150
USD  2021-06-02  GBP  1.415
ZAR  2021-06-02  GBP 19.422

[124 rows x 3 columns]

Upvotes: 1

Muhammad Hassan
Muhammad Hassan

Reputation: 4229

In case I understood you correctly, you want to operate over 100 rows at a time then sleep for 5 sec and continue again. Here is how you can do it:

from forex_python.converter import CurrencyRates
c = CurrencyRates()
for g_name, df_group in df.groupby(df.index//100):
    try:
        condition = df.index.isin(df_group.index)
        df['USD_rate'] = np.where(condition, df_group.apply(lambda x: c.get_rate(x['Currency'], x['Date']), axis = 1), np.nan)
        time.sleep(5)
    except:
        continue

Upvotes: 1

Related Questions