CODEWITHSUNDEEP
python-3.xpandasglobal-variables
Audrey
Audrey

Reputation: 43

Calling Pandas Data Frames Created with globals() Inside For Loop

I am iterating through 50 files in python and dumping them each into pandas data frames. Then from each data frame I create three new data frames based on the values in a specific field in the original data frame. These three new frames have new names that include the the value they were filtered on.

It works, yay! I get all my data frames!

The problem is, I'm creating these data frames using a global() call, and I do not know how to access them without explicitly typing each individual data frame name into a kernal.

Why do I want to do this, you may ask?

Well, I want to grab all of the data frames that end in 'cd', for example, and append (union all) them into a final data frame. I don't want to have to explicitly call all 50 of them. I want to loop through a list of the data frames to accomplish this task.

Any suggestions on how to accomplish this, or rework the code?

I'm new to these more intensive processes with iPython, so change whatever.

    filelist = os.listdir()
    sum_list = ['CAKE', 'TWINKIES', 'DOUGHNUTS', 'CUPCAKES']
    for f in filelist:
        state = re.match('((\w+){2})\_', f)
        state_df = str(state.group(1)) + '_df'
        data = pd.read_csv(f, low_memory = False)
        df = pd.DataFrame(data)
        for x in sum_list:
            sdo = state_df + '_' + x.lower()
            globals()[sdo] = pd.DataFrame(df.loc[df['summary_level'] == x])

Upvotes: 2

Views: 1096

Answers (1)

Andy Hayden
Andy Hayden

Reputation: 375595

I think a much better way is to create your own dictionary rather than resort to globals! Just create your own and append to some list or dictionary of lists? (depending on the classification):

dfs = {}
for f in filelist:
    ...
    df = pd.read_csv(f)  # this returns a DataFrame
    for x in sum_list:
        ...
        dfs[sdo] = df[df.summary_level == x]  # again, this return a DataFrame

You could use a default dict, and assign each to a sub dictionary:

from collection import defaultdict
dfs = defaultdict({})
...
        dfs[x][sdo] = ...

i.e. dfs['CAKE'] will be all the CAKE DataFrames.

Upvotes: 1

Related Questions