CODEWITHSUNDEEP
pythoncsvpickle
Nebula
Nebula

Reputation: 169

Processing multiple files and write a csv file for each

I wrote a code that works fine for single file, but I have to change the names for each file. It reads a pickle file, write it into a txt file, then does some process on the context of txt file and produce a list of numbers, at the end stores the list in a dataframe and write that dataframe in csv file.

def get_value_of_list(bit_list):
p_number = 0
for i in bit_list:
    if i == 1:
        p_number = p_number + 1
return p_number

def cross_entropy(p, q):
    return -sum([p[i] * log2(q[i]) for i in range(len(p))])

if __name__ == "__main__":

file_name = 'pickleData_AIMchat2.txt'
pickle_file = 'AIMchat2.pickle'
pk = PickleToFile(file_name, pickle_file)
pk.create_pickle_file()
h = HexToBinary(file_name)
hex_list = h.read_file()
num_of_bits = 8

scale = 16
bin_data = []
for i in hex_list:
    bin_data.append(bin(int(i, scale))[2:].zfill(num_of_bits))

my_bit_list = []
for byte in bin_data:
    bit_list = []
    for bit in byte:
        bit_list.append(int(bit))
    num_of_one_divided_by_eight = get_value_of_list(bit_list) / 8
    my_bit_list.append(num_of_one_divided_by_eight)

cross_entropy_list = []
i = 0
while i < len(my_bit_list):
    cross = cross_entropy([my_bit_list[i]], [my_bit_list[i + 1]])
    cross_entropy_list.append(cross)
    i = i + 2

df = pd.DataFrame(cross_entropy_list)
df.to_csv(r'AIMchat2.csv', index=False, index_label=False, chunksize=1000000, header=False)

I have changed create_pickle_file() to the code below to read files in the directory:

class PickleToFile:
    def __init__(self, name, pickle_file):
    self.name = name
    self.pickle_file = pickle_file

    def create_pickle_file(self):
    basepath = Path()
    files_in_basepath = basepath.iterdir('pickle/')
    for item in files_in_basepath:
        if item.is_file():
            checkThePickle = open(self.pickle_file, "rb")
            with open(self.name, 'w') as filehandler:
                for listItem in checkThePickle:
                    filehandler.write('%s\n' % listItem)

But since after reading file it writes it to a text file and then a csv file, I don't know how to do that. Appreciate any suggestions.

Upvotes: 0

Views: 270

Answers (1)

asylumax
asylumax

Reputation: 821

If you are looking to get a list of files in directory and process them, this should get you what you want:

How do I list all files of a directory?

Once you have this list of files, do a loop:

for each in list_of_files:
    process_function(each)

Then, you are on your way, where 'process_function' is the function, and the argument is the filename.

Upvotes: 1

Related Questions