CODEWITHSUNDEEP
python-3.xwindowsautomationjupyter-notebook
Shiva
Shiva

Reputation: 312

Automate the Script whenever a new folder/file is added in directory in Python

I have multiple folders in a directory and each folder has multiple files. I have a code which checks for a specific file in each folder and does some data preprocessing and analysis if the specific file is present. A snippet of it is given below.

import pandas as pd
import json
import os

rootdir = os.path.abspath(os.getcwd())

df_list = []

for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        if file.startswith("StudyParticipants") and file.endswith(".csv"):
            temp = pd.read_csv(os.path.join(subdir, file))
                           .....
                           ..... 
                          'some analysis'

Merged_df.to_excel(path + '\Processed Data Files\Study_Participants_Merged.xlsx')

Now, I want to automate this process. I want this script to be executed whenever a new folder is added. This is my first in exploring automation process and I ham stuck on this for quite a while without major progress.

I am using windows system and Jupyter notebook to create these dataframes and perform analysis.

Any help is greatly appreciated. Thanks.

Upvotes: 1

Views: 2233

Answers (2)

waykiki
waykiki

Reputation: 1094

I've wrote a script which you should only run once and it will work. Please note:

1.) This solution does not take into account which folder was created. If this information is required I can rewrite the answer.

2.) This solution assumes folders won't be deleted from the main folder. If this isn't the case, I can rewrite the answer as well.

import time
import os


def DoSomething():
    pass


if __name__ == '__main__':
    # go to folder of interest
    os.chdir('/home/somefolders/.../A1')
    # get current number of folders inside it
    N = len(os.listdir())
    while True:
        time.sleep(5)  # sleep for 5 secs
        if N != len(os.listdir()):
            print('New folder added! Doing something useful...')
            DoSomething()
            N = len(os.listdir())  # update N

Upvotes: 1

ScienceLover
ScienceLover

Reputation: 93

take a look at watchdog.

http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/

you could also code a very simple watchdog service on your own.

  1. list all files in the directory you want to observe
  2. wait a time span you define, say every few seconds
  3. make again a list of the filesystem
  4. compare the two lists, take the difference of them
  5. the resulting list from this difference are your filesystem changes

Best greetings

Upvotes: 0

Related Questions