CODEWITHSUNDEEP
pythonwindows
Alex
Alex

Reputation: 15

Duplicate in list created from filenames (python)

I'm trying to create a list of excel files that are saved to a specific directory, but I'm having an issue where when the list is generated it creates a duplicate entry for one of the file names (I am absolutely certain there is not actually a duplicate of the file).

import glob

# get data file names
path =r'D:\larvalSchooling\data'
filenames = glob.glob(path + "/*.xlsx")

output:

>>> filenames
['D:\\larvalSchooling\\data\\copy.xlsx', 'D:\\larvalSchooling\\data\\Raw data-SF_Fri_70dpf_GroupABC_n5_20200828_1140-Trial     1.xlsx', 'D:\\larvalSchooling\\data\\Raw data-SF_Sat_70dpf_GroupA_n5_20200808_1015-Trial     1.xlsx', 'D:\\larvalSchooling\\data\\Raw data-SF_Sat_84dpf_GroupABCD_n5_20200822_1440-Trial     1.xlsx', 'D:\\larvalSchooling\\data\\~$Raw data-SF_Fri_70dpf_GroupABC_n5_20200828_1140-Trial     1.xlsx']

you'll note 'D:\larvalSchooling\data\Raw data-SF_Fri_70dpf_GroupABC_n5_20200828_1140-Trial 1.xlsx' is listed twice.

Rather than going through after the fact and removing duplicates I was hoping to figure out why it's happening to begin with.

I'm using python 3.7 on windows 10 pro

Upvotes: 1

Views: 381

Answers (3)

codegs1514
codegs1514

Reputation: 36

You can use os.path.splittext to get the file extension and loop through the directory using os.listdir . The open excel files can be skipped using the following code:

filenames = []
for file in os.listdir('D:\larvalSchooling\data'):
    filename, file_extension = os.path.splitext(file)
    if file_extension == '.xlsx':
       if not file.startswith('~$'):
          filenames.append(file)

Note: this might not be the best solution, but it'll get the job done :)

Upvotes: 0

GaEsRaRe
GaEsRaRe

Reputation: 16

Whenever you open an excel file it will create a ghost copy that works as a temporary backup copy for that specific file. In this case:

Raw data-SF_Fri_70dpf_GroupABC_n5_20200828_1140-Trial1.xlsx

~$ Raw data-SF_Fri_70dpf_GroupABC_n5_20200828_1140-Trial1.xlsx

This means that the file is open by some software and it's showing you that backup inside(usually that file is hidden from the explorer as well)

Just search for the program and close it. Other actions, such as adding validation so the "~$.*.xlsx" type of file is ignored should be also implemented if this is something you want to avoid.

Upvotes: 0

Samwise
Samwise

Reputation: 71562

If you wrote the code to remove duplicates (which can be as simple as filenames = set(filenames)) you'd see that you still have two filenames. Print them out one on top of the other to make a visual comparison easier:

'D:\\larvalSchooling\\data\\Raw data-SF_Sat_84dpf_GroupABCD_n5_20200822_1440-Trial     1.xlsx',
'D:\\larvalSchooling\\data\\~$Raw data-SF_Fri_70dpf_GroupABC_n5_20200828_1140-Trial     1.xlsx'

The second one has a leading ~ (probably an auto-backup).

Upvotes: 1

Related Questions