Looping through files with Powershell after using System.IO.FileSystemWatcher

Question

Need some assistance with a PowerShell script that watches a folder for the creation of new files with a .csv extension.
When the creation of a new file is detected the following actions take place…

1. Details of the file are sent to a log file
2. A copy of the file is sent to another folder named “1_ToBeProcessed\MyImport.csv” (Note the file is renamed to MyImport.csv)
3. The file is moved to the directory “\MYSERVER\MYCOMPANY\MYFOLDER\MyHistory” (With its original file name)
4. An expression is invoked to run a .bat file which will take the file located in “\MYSERVER\MYCOMPANY\MYFOLDER\1_ToBeProcessed” and import the file into an application.
5. The script then sleeps for 60 seconds to allow the import to complete before processing the next file. This script all works great unless another file is created while the script is running.
   Multiple files of the same type might hit the watch folder at the same time. Therefore, the script could be looping through five files and then another file could come into the directory. When that happens, it appears the script starts over again and begins looping through the files…but it keeps looping the same files over and over again. I am hoping that someone can point me in the right direction as to what I am doing wrong with this script. How do I get it to complete the files it is working on in the current loop before starting again because a new file was added? Any assistance that could be provided would be greatly appreciated!

This is what I currently have...

#Define watcher; Set path to watch; Set filter on file type
$filewatcher = New-Object System.IO.FileSystemWatcher
$filewatcher.Path = "\\MYSERVER\MYCOMPANY\MYFOLDER"
$filewatcher.Filter = "*.csv"                     

#Define "move to" and "copy to" paths
$moveTo = "\\MYSERVER\MYCOMPANY\MYFOLDER\MyHistory"
$copyTo = "\\MYSERVER\MYCOMPANY\MYFOLDER\1_ToBeProcessed\MyImport.csv"


#Define actions after an Event is Detected
$action = {$files = Get-ChildItem -Path $filewatcher.Path -Filter filewatcher.Filter 
        foreach ($file in $files)
        {
        #Define variables for log file
        $changeType = $Event.SourceEventArgs.ChangeType
        $logline = "$(Get-Date), $changeType, $file"
        #Actions to take
        Add-Content "\\MYSERVER\MYCOMPANY\MYFOLDER\3_Log\MyImportLog.txt" -value $logline
        Copy-Item $file.FullName -Destination $copyTo
        Move-Item $file.FullName -Destination $moveTo -Force
        Invoke-Expression "& '\\MYSERVER\MYCOMPANY\MYFOLDER\4_Scripts\PSscriptToRunBATfile.ps1'"
        #Pause the script for 60 seconds to allow it to finish posting the import before going to the next record
        Start-Sleep -Seconds 60
        }
        }

#Decide which events should be watched and set check frequency 
$onCreated = Register-ObjectEvent -InputObject $filewatcher -EventName "Created" -SourceIdentifier FileCreated -Action $action

Answer 1

I think you might be getting some sort of file lock issue, but it's hard to tell without knowing what's in your PSscriptToRunBATfile.ps1 file.

Your filewatcher fires on the default NotifyEvents of LastWrite | FileName | DirectoryName and you loop the directory at that point. However, your get-childitem command is not necessarily returning the files in the same order that they are being copied in.

A better solution would have your action to do work based on your $Event object and change your PS1 file take the file that caused the event as an input parameter:

$action = {$file = $Event.SourceEventArgs.FullPath
  #Define variables for log file
  $changeType = $Event.SourceEventArgs.ChangeType
  $logline = "$(Get-Date), $changeType, $file"
  #Actions to take
  Add-Content "\\MYSERVER\MYCOMPANY\MYFOLDER\3_Log\MyImportLog.txt" -value $logline
  Copy-Item $file -Destination $copyTo
  Move-Item $file -Destination $moveTo -Force
  Invoke-Expression "& '\\MYSERVER\MYCOMPANY\MYFOLDER\4_Scripts\PSscriptToRunBATfile.ps1' -InputFile $file"
}

Note: You may want to increase the InternalBufferSize above the default 8k so you don't miss any changes:

    $filewatcher.InternalBufferSize = 32kb

Answer 2

I'd suggest not looping, rather processing the files as they are created/changed. Maybe if you have lots of files to start with, you either:

(a) loop first (and once only). Then enable the file watcher to act upon which ever files have changed after that.

(b) enable the file watcher, with FileChanged & FileCreated, then 'touch' all the files to fire off your code.

# based on the script by By BigTeddy 05 September 2011
# simplified by Andy Myatt, to use one script block
# https://gallery.technet.microsoft.com/scriptcenter/Powershell-FileSystemWatche-dfd7084b

param(
    [string]$folderToWatch = "D:\Temp"
  , [string]$filter        = "*.*"
  , [string]$logFile       = "D:\Temp\Temp2\filewatcher.log"
)

# In the following line, you can change 'IncludeSubdirectories to $true if required.
$fsw = New-Object IO.FileSystemWatcher $folderToWatch, $filter -Property @{IncludeSubdirectories = $false;NotifyFilter = [IO.NotifyFilters]'FileName, LastWrite'}

# This script block is used/called by all 3 events and:
# appends the event to a log file, as well as reporting the event back to the console
$scriptBlock = {

  # REPLACE THIS SECTION WITH YOUR PROCESSING CODE
  $logFile = $event.MessageData # message data is how we pass in an argument to the event script block
  $name = $Event.SourceEventArgs.Name
  $changeType = $Event.SourceEventArgs.ChangeType
  $timeStamp = $Event.TimeGenerated
  Write-Host "$timeStamp|$changeType|'$name'" -fore green
  Out-File -FilePath $logFile -Append -InputObject "$timeStamp|$changeType|'$name'"
  # REPLACE THIS SECTION WITH YOUR PROCESSING CODE

}

# Here, all three events are registered.  You need only subscribe to events that you need:
Register-ObjectEvent $fsw Created -SourceIdentifier FileCreated -MessageData $logFile -Action $scriptBlock
Register-ObjectEvent $fsw Deleted -SourceIdentifier FileDeleted -MessageData $logFile -Action $scriptBlock
Register-ObjectEvent $fsw Changed -SourceIdentifier FileChanged -MessageData $logFile -Action $scriptBlock

# To stop the monitoring, run the following commands:
#  Unregister-Event FileDeleted  ;  Unregister-Event FileCreated  ;  Unregister-Event FileChanged


#This script uses the .NET FileSystemWatcher class to monitor file events in folder(s).
#The advantage of this method over using WMI eventing is that this can monitor sub-folders.
#The -Action parameter can contain any valid Powershell commands.
#The script can be set to a wildcard filter, and IncludeSubdirectories can be changed to $true.
#You need not subscribe to all three types of event.  All three are shown for example.