Python - run function when folder changes

Question

I want to run my function whenever a change occurs in my folder. The main thing to do here is to normalize new files, we do so by comparing my two folders, the folder that contains the normalized ones and the folder where we add new files and return a list of the unmatched files which our normalize function will modify and place in the folder of normalized ones.

Here is my code:

def normalize(c):
    PATH = path_to_file 
    fileNames = os.listdir(PATH)

    fileNames = c
    for file in fileNames:
        sfm = pd.read_csv(PATH + file, delimiter = ';', parse_dates=[[0, 1]], skiprows = 3)
  # other instructions then rewrite my new dataframe in my normalized folder
sfm.to_csv('.\dirnormalized'+str(file[:-4])+'.csv', sep=';', index=False, header=True)

def comparison():
    PATH = path_to_file 
    PATH_norm = path_to_norm
    fileNames = os.listdir(PATH)
    files = os.listdir(PATH_norm)
    fileNames = [file for file in fileNames if '.csv' in file]
    files = [fl for fl in files if '.csv' in fl]
    diff = [x for x in fileNames if x not in files]
    return diff

def change():

    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')
# path_to_file and word_file are the same the only difference is that in path_to_file we have a back slash in the end in order to acces the files one by one 
    path = word_file
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

def process(): 
    if(change()):
        c = comparison()
        normalize(c)
if __name__ == "__main__":
    import os 
    import time
    import logging
    from watchdog.observers import Observer
    from watchdog.events import LoggingEventHandler

    process()

The only problem here is that in my process() function the if() instruction doesn't seem pass to other ones, I can see when the new files are added which means when a modification occurs but the other instructions don't occur it stucks on if().

Answer 1

Check out this for changes in a dir.

Use the MS ReadDirectoryChanges API, exposed via the pywin32 win32file module.

The way we employ it here is to use call ReadDirectoryChangesW in blocking mode.

The function returns a list of 2-tuples, each one representing an action and a filename. A rename always gives a pair of 2-tuples; other compound actions may also give a list.

import os
import win32file
import win32con

ACTIONS = {
  1 : "Created",
  2 : "Deleted",
  3 : "Updated",
  4 : "Renamed from something",
  5 : "Renamed to something"
}
# Thanks to Claudio Grondi for the correct set of numbers
FILE_LIST_DIRECTORY = 0x0001

path_to_watch = "."
hDir = win32file.CreateFile (
  path_to_watch,
  FILE_LIST_DIRECTORY,
  win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE | win32con.FILE_SHARE_DELETE,
  None,
  win32con.OPEN_EXISTING,
  win32con.FILE_FLAG_BACKUP_SEMANTICS,
  None
)
while 1:
  #
  # ReadDirectoryChangesW takes a previously-created
  # handle to a directory, a buffer size for results,
  # a flag to indicate whether to watch subtrees and
  # a filter of what changes to notify.
  #
  # NB Tim Juchcinski reports that he needed to up
  # the buffer size to be sure of picking up all
  # events when a large number of files were
  # deleted at once.
  #
  results = win32file.ReadDirectoryChangesW (
    hDir,
    1024,
    True,
    win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
     win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
     win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
     win32con.FILE_NOTIFY_CHANGE_SIZE |
     win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
     win32con.FILE_NOTIFY_CHANGE_SECURITY,
    None,
    None
  )
  for action, file in results:
    full_filename = os.path.join (path_to_watch, file)
    print full_filename, ACTIONS.get (action, "Unknown")

With some editing, this could easily achieve what you want, as the winApi is leveraged, its likely to be less error prone.

Thanks Tim Golden.

Answer 2

The last line of process() namely observer.join() will probably block the thread preventing the function from returning.

Moreover, control leave the function returning None causing the if statement to never be executed.