CODEWITHSUNDEEP
pythonpandas
nickolasclarke
nickolasclarke

Reputation: 99

updating dict value does not update global var it references

I am attempting to iterate over a dict of dataframes, modify them with a function, and then assign the returned dfs to their global vars. I would expect any value in a key-value pair of a dict to be a pointer to the variable passed. Instead, it appears to be updating only the value in the data dict. This is unexpected. What am I misunderstanding about identifiers? I found this question asked the same thing in the second half, but I did not understand the accepted answer.

Please see my demonstration below:

import pandas as pd

bids = pd.read_csv('data/as_bid_aggregated_data.csv')
plans = pd.read_csv('data/as_plan.csv')
energy_prices = pd.read_csv('data/as_bid_aggregated_data.csv')
price_vol = pd.read_csv('data/as_price_vol.csv')
generation = pd.read_csv('data/generation.csv')

data = {'bids':bids,
        'plans':plans,
        'energy_prices':energy_prices,
        'price_vol':price_vol,
        'generation':generation,
       }

I evaluate bids to demonstrate what it looks initially like after import: 

bids.head().to_clipboard()
 OUTPUT: 
 note the index, date, and hr_beg cols. These should be modified for all dfs in data after processing. 
 V  V          V
    date          hr_beg    OFFNS_Unweighted Average Price  OFFNS_Max Price OFFNS_Min Price OFFNS_Total Quantity    OFFNS_Number of Bids    OFFNS_Weighted Avg Price    ONNS_Unweighted Average Price   ONNS_Max Price  ONNS_Min Price  ONNS_Total Quantity ONNS_Number of Bids ONNS_Weighted Avg Price REGDN_Unweighted Average Price  REGDN_Max Price REGDN_Min Price REGDN_Total Quantity    REGDN_Number of Bids    REGDN_Weighted Avg Price    REGUP_Unweighted Average Price  REGUP_Max Price REGUP_Min Price REGUP_Total Quantity    REGUP_Number of Bids    REGUP_Weighted Avg Price    RRSGN_Unweighted Average Price  RRSGN_Max Price RRSGN_Min Price RRSGN_Total Quantity    RRSGN_Number of Bids    RRSGN_Weighted Avg Price    RRSNC_Unweighted Average Price  RRSNC_Max Price RRSNC_Min Price RRSNC_Total Quantity    RRSNC_Number of Bids    RRSNC_Weighted Avg Price
# 0 2014-01-01  0   43.3190909090909    300.01  0.01    38144.7 22  59.51279016481975   22.016969696969696  250.0   1.0 32531.499999999985  33  36.74238980680264   20.669076923076922  500.0   0.92    71971.59999999992   65  26.577483215601717  19.744255319148944  500.0   0.01    56916.80000000003   47  27.33264099527731   20.85708333333334   500.0   0.01    107723.6    48  30.19552034094665   1.5 3.0 0.0 2236.8  2   1.5996512875536482
# 1 2014-01-01  1   43.342727272727274  300.01  0.01    38216.4 22  59.505340220428934  20.93514285714285   250.0   1.0 34781.19999999998   35  34.95683860821363   21.764761904761905  500.0   0.8 70412.39999999994   63  27.92263442234607   18.834375000000012  500.0   0.01    50201.80000000002   48  28.87979570453649   19.6692 500.0   0.01    107145.0    50  30.00068717158991   1.5 3.0 0.0 2235.8  2   1.599695858305752
# 2 2014-01-01  2   43.34818181818181   300.01  0.01    38336.9 22  59.49848289767822   20.97   250.0   1.0 34741.39999999999   35  35.091575987150776  21.836461538461545  500.0   0.58    72212.29999999992   65  28.27043938498013   18.856041666666666  500.0   0.01    50769.90000000001   48  28.61359006025224   19.5252 500.0   0.01    105503.8    50  30.27549695840339   1.5 3.0 0.0 2236.2  2   1.5996780252213578
# 3 2014-01-01  3   43.35000000000001   300.01  0.01    38374.5 22  59.492013316134425  21.00257142857142   250.0   1.0 34761.399999999994  35  35.11167079001421   22.38730158730159   500.0   0.53    70801.39999999994   63  28.66950969896075   18.854583333333334  500.0   0.01    50313.10000000001   48  28.865852233314985  19.5298 500.0   0.01    105024.0    50  30.41884454981718   1.5 3.0 0.0 2238.2  2   1.5995889554105982
# 4 2014-01-01  4   46.431  300.01  0.01    33460.8 20  64.00475684980633   20.75628571428571   250.0   1.0 34829.29999999999   35  34.791386648597594  21.684531250000006  500.0   0.7 71841.29999999992   64  27.846364904309922  19.238510638297864  500.0   0.01    50767.90000000001   47  28.70213516808849   19.801836734693875  500.0   0.01    104199.79999999996  49  30.477332029428077  1.5 3.0 0.0 2242.4  2   1.5994024259721726

I then create a function to modify a given dataframe. It combines cols to create a single datetime index, and replace the index with that, I've redacted the logic for simplicity.

def create_dt(input_df):
    '''create a dataframe with a datetime index from multiple cols
    '''
    df = input_df.copy()
    #modify the df
    df = df.set_index(dt_index)
    df = df.drop(columns=[date_col,hr_col])
    return df

I then attempt to unpack data, pass them into create_dt() and assign the results. I would expect this to update the global var for each df, via the pointer in the dict. 

for key, df in data.items():
    data[key] = create_dt(data[key],'date','hr_beg')

I evaluate bids global, post function call. It remains the same. 

# OUTPUT:
bids.head().to_clipboard()
# note the index, date, and hr_beg cols. Same as initial value
# V  V          V
#       date    hr_beg  OFFNS_Unweighted Average Price  OFFNS_Max Price OFFNS_Min Price OFFNS_Total Quantity    OFFNS_Number of Bids    OFFNS_Weighted Avg Price    ONNS_Unweighted Average Price   ONNS_Max Price  ONNS_Min Price  ONNS_Total Quantity ONNS_Number of Bids ONNS_Weighted Avg Price REGDN_Unweighted Average Price  REGDN_Max Price REGDN_Min Price REGDN_Total Quantity    REGDN_Number of Bids    REGDN_Weighted Avg Price    REGUP_Unweighted Average Price  REGUP_Max Price REGUP_Min Price REGUP_Total Quantity    REGUP_Number of Bids    REGUP_Weighted Avg Price    RRSGN_Unweighted Average Price  RRSGN_Max Price RRSGN_Min Price RRSGN_Total Quantity    RRSGN_Number of Bids    RRSGN_Weighted Avg Price    RRSNC_Unweighted Average Price  RRSNC_Max Price RRSNC_Min Price RRSNC_Total Quantity    RRSNC_Number of Bids    RRSNC_Weighted Avg Price
# 0 2014-01-01  0   43.3190909090909    300.01  0.01    38144.7 22  59.51279016481975   22.016969696969696  250.0   1.0 32531.499999999985  33  36.74238980680264   20.669076923076922  500.0   0.92    71971.59999999992   65  26.577483215601717  19.744255319148944  500.0   0.01    56916.80000000003   47  27.33264099527731   20.85708333333334   500.0   0.01    107723.6    48  30.19552034094665   1.5 3.0 0.0 2236.8  2   1.5996512875536482
# 1 2014-01-01  1   43.342727272727274  300.01  0.01    38216.4 22  59.505340220428934  20.93514285714285   250.0   1.0 34781.19999999998   35  34.95683860821363   21.764761904761905  500.0   0.8 70412.39999999994   63  27.92263442234607   18.834375000000012  500.0   0.01    50201.80000000002   48  28.87979570453649   19.6692 500.0   0.01    107145.0    50  30.00068717158991   1.5 3.0 0.0 2235.8  2   1.599695858305752
# 2 2014-01-01  2   43.34818181818181   300.01  0.01    38336.9 22  59.49848289767822   20.97   250.0   1.0 34741.39999999999   35  35.091575987150776  21.836461538461545  500.0   0.58    72212.29999999992   65  28.27043938498013   18.856041666666666  500.0   0.01    50769.90000000001   48  28.61359006025224   19.5252 500.0   0.01    105503.8    50  30.27549695840339   1.5 3.0 0.0 2236.2  2   1.5996780252213578
# 3 2014-01-01  3   43.35000000000001   300.01  0.01    38374.5 22  59.492013316134425  21.00257142857142   250.0   1.0 34761.399999999994  35  35.11167079001421   22.38730158730159   500.0   0.53    70801.39999999994   63  28.66950969896075   18.854583333333334  500.0   0.01    50313.10000000001   48  28.865852233314985  19.5298 500.0   0.01    105024.0    50  30.41884454981718   1.5 3.0 0.0 2238.2  2   1.5995889554105982
# 4 2014-01-01  4   46.431  300.01  0.01    33460.8 20  64.00475684980633   20.75628571428571   250.0   1.0 34829.29999999999   35  34.791386648597594  21.684531250000006  500.0   0.7 71841.29999999992   64  27.846364904309922  19.238510638297864  500.0   0.01    50767.90000000001   47  28.70213516808849   19.801836734693875  500.0   0.01    104199.79999999996  49  30.477332029428077  1.5 3.0 0.0 2242.4  2   1.5994024259721726

I then evaluate the bids dataframe k-v pair in data. It is successfully modified.

data['bids'].head().to_clipboard()
#OUTPUT
# note datetime index, no date or hr_beg cols, see .columns() output one cell below. 
# V
#   OFFNS_Unweighted Average Price  OFFNS_Max Price OFFNS_Min Price OFFNS_Total Quantity    OFFNS_Number of Bids    OFFNS_Weighted Avg Price    ONNS_Unweighted Average Price   ONNS_Max Price  ONNS_Min Price  ONNS_Total Quantity ONNS_Number of Bids ONNS_Weighted Avg Price REGDN_Unweighted Average Price  REGDN_Max Price REGDN_Min Price REGDN_Total Quantity    REGDN_Number of Bids    REGDN_Weighted Avg Price    REGUP_Unweighted Average Price  REGUP_Max Price REGUP_Min Price REGUP_Total Quantity    REGUP_Number of Bids    REGUP_Weighted Avg Price    RRSGN_Unweighted Average Price  RRSGN_Max Price RRSGN_Min Price RRSGN_Total Quantity    RRSGN_Number of Bids    RRSGN_Weighted Avg Price    RRSNC_Unweighted Average Price  RRSNC_Max Price RRSNC_Min Price RRSNC_Total Quantity    RRSNC_Number of Bids    RRSNC_Weighted Avg Price
# 2014-01-01 00:00:00   43.3190909090909    300.01  0.01    38144.7 22  59.51279016481975   22.016969696969696  250.0   1.0 32531.499999999985  33  36.74238980680264   20.669076923076922  500.0   0.92    71971.59999999992   65  26.577483215601717  19.744255319148944  500.0   0.01    56916.80000000003   47  27.33264099527731   20.85708333333334   500.0   0.01    107723.6    48  30.19552034094665   1.5 3.0 0.0 2236.8  2   1.5996512875536482
# 2014-01-01 01:00:00   43.342727272727274  300.01  0.01    38216.4 22  59.505340220428934  20.93514285714285   250.0   1.0 34781.19999999998   35  34.95683860821363   21.764761904761905  500.0   0.8 70412.39999999994   63  27.92263442234607   18.834375000000012  500.0   0.01    50201.80000000002   48  28.87979570453649   19.6692 500.0   0.01    107145.0    50  30.00068717158991   1.5 3.0 0.0 2235.8  2   1.599695858305752
# 2014-01-01 02:00:00   43.34818181818181   300.01  0.01    38336.9 22  59.49848289767822   20.97   250.0   1.0 34741.39999999999   35  35.091575987150776  21.836461538461545  500.0   0.58    72212.29999999992   65  28.27043938498013   18.856041666666666  500.0   0.01    50769.90000000001   48  28.61359006025224   19.5252 500.0   0.01    105503.8    50  30.27549695840339   1.5 3.0 0.0 2236.2  2   1.5996780252213578
# 2014-01-01 03:00:00   43.35000000000001   300.01  0.01    38374.5 22  59.492013316134425  21.00257142857142   250.0   1.0 34761.399999999994  35  35.11167079001421   22.38730158730159   500.0   0.53    70801.39999999994   63  28.66950969896075   18.854583333333334  500.0   0.01    50313.10000000001   48  28.865852233314985  19.5298 500.0   0.01    105024.0    50  30.41884454981718   1.5 3.0 0.0 2238.2  2   1.5995889554105982
# 2014-01-01 04:00:00   46.431  300.01  0.01    33460.8 20  64.00475684980633   20.75628571428571   250.0   1.0 34829.29999999999   35  34.791386648597594  21.684531250000006  500.0   0.7 71841.29999999992   64  27.846364904309922  19.238510638297864  500.0   0.01    50767.90000000001   47  28.70213516808849   19.801836734693875  500.0   0.01    104199.79999999996  49  30.477332029428077  1.5 3.0 0.0 2242.4  2   1.5994024259721726

data['bids'].columns()

#OUTPUT:
# Index(['OFFNS_Unweighted Average Price', 'OFFNS_Max Price', 'OFFNS_Min Price',
#        'OFFNS_Total Quantity', 'OFFNS_Number of Bids',
#        'OFFNS_Weighted Avg Price', 'ONNS_Unweighted Average Price',
#        'ONNS_Max Price', 'ONNS_Min Price', 'ONNS_Total Quantity',
#        'ONNS_Number of Bids', 'ONNS_Weighted Avg Price',
#        'REGDN_Unweighted Average Price', 'REGDN_Max Price', 'REGDN_Min Price',
#        'REGDN_Total Quantity', 'REGDN_Number of Bids',
#        'REGDN_Weighted Avg Price', 'REGUP_Unweighted Average Price',
#        'REGUP_Max Price', 'REGUP_Min Price', 'REGUP_Total Quantity',
#        'REGUP_Number of Bids', 'REGUP_Weighted Avg Price',
#        'RRSGN_Unweighted Average Price', 'RRSGN_Max Price', 'RRSGN_Min Price',
#        'RRSGN_Total Quantity', 'RRSGN_Number of Bids',
#        'RRSGN_Weighted Avg Price', 'RRSNC_Unweighted Average Price',
#        'RRSNC_Max Price', 'RRSNC_Min Price', 'RRSNC_Total Quantity',
#        'RRSNC_Number of Bids', 'RRSNC_Weighted Avg Price'],
#       dtype='object')

Upvotes: 2

Views: 279

Answers (2)

Phillyclause89
Phillyclause89

Reputation: 680

If you want to change the pandas.DataFrame object and have it updated for all variables pointed to that object then you need to use the inplace=True agrument for all the df method calls you use. Step through these examples in python tutor to get a clearer idea of what objects are being pointed to by your variables:

import pandas as pd
'''Basicly What you were doing'''
def create_dt(input_df):
    df = input_df.copy()
    df = df.set_index(pd.Series(['i','j']))
    return df


x = pd.DataFrame({'a':[1,2],'b':[3,4]})
datax = {'x':x,}
for key, df in datax.items():
    datax[key] = create_dt(datax[key])
print(x)


'''Basicly What was recommended'''
def create_dt2(input_df):
    input_df = input_df.set_index(pd.Series(['i','j']))
    return input_df


y = pd.DataFrame({'a':[5,6],'b':[7,8]})
datay = {'y':y,}
for key, df in datay.items():
    datay[key] = create_dt2(datay[key])

print(y)

'''Using inplace = True is the only way to change the object'''
def modify_df(input_df):
    input_df.set_index(pd.Series(['i','j']), inplace=True)

z = pd.DataFrame({'a':[9,10],'b':[11,12]})
dataz = {'z':z,} 
for key, df in dataz.items():
    modify_df(dataz[key])

print(z)

Upvotes: 1

monkut
monkut

Reputation: 43840

In the global scope you define a number of dataframes:

bids = pd.read_csv('data/as_bid_aggregated_data.csv')
plans = pd.read_csv('data/as_plan.csv')
energy_prices = pd.read_csv('data/as_bid_aggregated_data.csv')
price_vol = pd.read_csv('data/as_price_vol.csv')
generation = pd.read_csv('data/generation.csv')

Then you create a dictionary of the following keys and assign the values to the above dataframes:


data = {'bids':bids,
        'plans':plans,
        'energy_prices':energy_prices,
        'price_vol':price_vol,
        'generation':generation,
       }

At this point your keys point to the dataframes in the outer scope.

Then you call a function that creates a COPY of the source dataframe, modifies it and returns it. 

def create_dt(input_df):
    '''create a dataframe with a datetime index from multiple cols
    '''
    df = input_df.copy()
    #modify the df
    df = df.set_index(dt_index)
    df = df.drop(columns=[date_col,hr_col])
    return df

for key, df in data.items():
    data[key] = create_dt(data[key],'date','hr_beg')

At this point the returned df from create_dt() is a different dataframe from the one given (you created a copy), and changed the reference in the dictionary, data. So there is no reason why the outer scope dataframes should be modified. (if you remove the input_df.copy() line it may work as your expecting)

In anycase, if this is all you are doing in the function there is no reason not do perform the drop columns in the loop from the outer scope, if you *want* the results to be modified.

Upvotes: 1

Related Questions