CODEWITHSUNDEEP
pythonloopscsvrecursionmerge
Kevy
Kevy

Reputation: 21

looping through folder of csvs python

I have been looking for sometime now and I have not had any luck. Here is my issue: I have a network drive filled with folders with sub-folders of CSVs. Eventually, these csvs need to get imported into a database. Based on the structure there is one row (the second line of each sheet) that I want removed from each one and appended to one new sheet from each other one to create its own sheet and table. A while back I found out that Python can achieve this. However, I ran into some issues. I am doing this one step at a time, so I do not feel overwhelmed by not knowing where to start. The problem is that I find all of the CSVs, but I cannot open each one to read any line to work on writing to a file.. I have been using some other threads as resources, but ran into IOError: [Errno 13] Permission denied: '.' I tried to exhaust all of my options before I came here, but now I am running out of time. I would more than appreciate the help.

Here is the code and as you can see from the comments that I have been playing for a while: 

#!/usr/bin/python
import os
import csv
import sys

#output_file = sys.argv[1]
input_path = sys.argv[1] #I would pass a '.' here for current directory on the drive
#output_file = sys.argv[2]


def doWhatYouWant(line):
    print line
    return line 
    #let the function return, not only print, to get the value for use as below 

#filewriter = csv.writer(open(output_file,'wb'))
#This recursively opens opens .csv files and opens them

directory = os.path.join(input_path)
for root,dirs,files in os.walk(directory):
    for file in files:
       if file.endswith(".csv"):
           f=open(input_path, 'r')
           lines= f.readlines()
           f.close()

           #reader =csv.DictReader(f,delimiter=',')
           # writer = open("testsummary.txt",'wb')
           # writer = csv.writer(writer, delimiter=',')

           f=open(file.txt,'w')

           #for row in reader:
           #        writer.writerow(row[2])
            #   print(row[1])

           newline=doWhatYouWant(line)
           f.write(newline)
           f.close()


           #f.close()
           #print file

Thank you all for your help in advance.

Upvotes: 0

Views: 932

Answers (3)

Kevy
Kevy

Reputation: 21

I received some help and did some additional research and got the code working. Thanks for all of your help. I believe I have it working to where it is manageable for me. It will go through all folders in a directory read the csv for its second row that I want, and will write all of the results to one sheet.

#!/usr/bin/python

#imported modules that are used

import csv
import glob
import os
import sys
import linecache


# Where the two arguments that are passed when running the program are stored.   
input_path = sys.argv[1] #input parameter accepted when running the program
output_file = sys.argv[2] #output parameter is the name of the our put file 

#The stored header for the .csv
header = [] #Where I store the header to be printed later

#Opens the output folder that will be written to with the .csv extension
with open(output_file+'.csv','wb') as outfile:
    filewriter = csv.writer(outfile)
#The header written here from the list created above ^
    filewriter.writerow(header)

#The loop for the files found from the specified input that are named with a date and eyepiece information for a name
    for input_file in glob.glob(os.path.join(input_path,'*/[0-2]*.csv')):
        with open(input_file,'rU') as csv_file:
            print "reading {:s}".format(input_file) #prints the names of the files processed to the console for a quick verification
            reader_hopefully = csv.reader(csv_file, delimiter=',', quoting=csv.QUOTE_NONE)
            next(reader_hopefully,None) #skips per line in the csv being read
            row=next(reader_hopefully,None) #the row that is actually necessary stored in a variable
            filewriter.writerow(row) #Writes necessary row to csv
            csv_file.close() #Closes open csv when finished reading it
    outfile.close()

Upvotes: 0

PM 2Ring
PM 2Ring

Reputation: 55469

You are getting the IOError: [Errno 13] Permission denied: '.' exception because you are attempting to open the current directory itself as if it were a readable text file:

open(input_path, 'r')

Instead, you need to do something like this:

open(os.path.join(root, file), 'r')

Also consider using with when opening files. Eg

with open(filename, 'r') as f:

Upvotes: 1

jayme
jayme

Reputation: 1316

You will have to compute the path to your file like:

for root, dirs, files in os.walk(directory):
    for file in files:
        if file.endswith(".csv"):
            f = open(os.path.join(root, file), 'r')

Upvotes: 0

Related Questions