CODEWITHSUNDEEP
pythoncsvpandas
Stefano Potter
Stefano Potter

Reputation: 3577

Applying same code to multiple files within multiple folders

I have a folder with numerous subdirectories (more folders) and in each subdirectory are csv files. I want to apply the same code to all the csv files in the subdirectories. If I did this for just one folder I would do it like this:

list1=[]  
pth=r'G:\Stefano\Ecoregion_assessment\csv_by_ecoregion_crp\05f08_46e'   
for f in os.listdir(pth):
    out=r'G:\Stefano\Ecoregion_assessment\final_files'
    df=pd.read_csv(os.path.join(pth,f))
    columns=['Percent', 'Land_Use', 'LC_Source']
    df=df[columns]
    df['Land_Use2']=df.Land_Use
    df.rename(columns={'Percent': 'Percent_' +df.iloc[1,2], 'Land_Use': 'Land_Use_' +df.iloc[1,2]} , inplace=True)
    df.drop(['LC_Source'], inplace=True, axis=1)
    list1.append(df)
    df_final = reduce(lambda left,right: pd.merge(left,right,on=['Land_Use2'], how='outer'), list1)
df_final.to_csv(os.path.join(out,'05f08_46e.csv'))

in this case G:\Stefano\Ecoregion_assessment\csv_by_ecoregion_crp is the root which navigates to all the sub directories and 05f08_46e is one of the sub directories. I want to apply this same code though to all of the folders within the root using a function and then send the df_final file to out and the name of the particular sub directory that is being looped through. I have 20 folders within G:\Stefano\Ecoregion_assessment\csv_by_ecoregion_crp and therefore I would like to have 20 output files to G:\Stefano\Ecoregion_assessment\final_files at the end. I simply want to apply the code I have written to all 20 folders without manually changing the folder pathways.

Possible another way to approach this would be to use os.walk but I have been playing around with it with no success yet.

Upvotes: 0

Views: 82

Answers (1)

Jean-François Fabre
Jean-François Fabre

Reputation: 140186

Just add an extra loop. I have tried to rewrite your code, even if some parts are missing and I cannot test it but I'm quite confident about it:

pth=r'G:\Stefano\Ecoregion_assessment\csv_by_ecoregion_crp'  # upper dir  
out=r'G:\Stefano\Ecoregion_assessment\final_files' # out of the loop

for d in os.listdir(pth):
    # 05f08_46e will be one of the "d" values
    for f in os.listdir(os.path.join(pth,d)):
      df=pd.read_csv(os.path.join(pth,f))
      columns=['Percent', 'Land_Use', 'LC_Source']
      df=df[columns]
      df['Land_Use2']=df.Land_Use
      df.rename(columns={'Percent': 'Percent_' +df.iloc[1,2], 'Land_Use': 'Land_Use_' +df.iloc[1,2]} , inplace=True)
      df.drop(['LC_Source'], inplace=True, axis=1)
      list1.append(df)
      df_final = reduce(lambda left,right: pd.merge(left,right,on=['Land_Use2'], how='outer'), list1)
    df_final.to_csv(os.path.join(out,d+'.csv'))

Upvotes: 1

Related Questions