CODEWITHSUNDEEP
pythonlistloopsdictionaryif-statement
TangerCity
TangerCity

Reputation: 845

How to process multiple files based on their names?

With below code Im collecting files from multiple directories. I check their dates, check if they are present in list missing_dates and I check if that specific date is present in every directory. After that, I process those files with the same date to 1 file.

Short said: 3 files of same date in every directory gets proccesed to 1 file.

This is my code:

missing_dates = ['20200907', '20200908', '20200909']

root=Path(r'c:\data\FF\Desktop\new_location\middle_stage_preprocessed_district')

data_per_date = dict()
for missing_date in missing_dates:
    print(f"processing {missing_date}")
    files=[fn for fn in (e for e in root.glob(f"**/*_{missing_date}.txt") if e.is_file())]
    if len(files) != 3:
        # stop processing, check other date
        continue

    for file in files:
        with open(file, 'r') as log_file:
            reader = csv.reader(log_file, delimiter = ',')
            next(reader) # skip header
            for row in reader:
                if filter_row(row):
                    vehicle_loc_dict[(row[9], location_token(row))].append(row)

    data_per_date[missing_date] = vehicle_loc_dict  
                                                                                             
for date in missing_dates:
    file_name = "MM{}-AB.dat".format(date)
    full_path = os.path.join(my_files, 'Directory_X', file_name)
    with open(full_path, 'w+') as output:
        writer = csv.writer(output, delimiter = '\t')
        writer.writerow(headers)
        writer.writerow(data)
        vehicle_loc = data_per_date[date]
        for vehicle_loc_list in vehicle_loc.values():
            for record_group in group_records(vehicle_loc_list):
                writer.writerow(output_record(record_group))

My file structure is like this:

├── dir_1
│   ├── XX_20200907.txt
│   └── XX_20200908.txt
├── dir_2
│   ├── YY_20200907.txt
│   └── YY_20200908.txt
└── dir_3
    ├── ZZ_20200907.txt
    └── ZZ_20200908.txt

I receive below error, but I dont know why.

Traceback (most recent call last):

  File "C:\data\FF\Desktop\Python\Python\Official_part1.py", line 271, in <module>
    vehicle_loc = data_per_date[date]

KeyError: '20200909'

Upvotes: 0

Views: 49

Answers (1)

Leos313
Leos313

Reputation: 5627

Simply the keyword 20200909 is not present in your dictionary. You can add a control mechanism that catch your error and through a warning, an error, an exception. There are many ways of solving your problem. The question for you is: what do you want to do? exit with an error? continuing processing?

Upvotes: 1

Related Questions