exclude path from get-childitem search

Question

I'm trying to have all the files in the folder structure copied to a folder that is also part of that structure. So the destination folder is excluded form the search. I also want to exclude any folder that has in its path ".thumbnails" but when I replace the full path in the $Skip with a wild card path such as 'D:\ZZZ_Phone_test*.thumbnails' it won't work.

Secondly, I'd like to make this more efficient if possible so the job can be finished quicker. When the script is running it is mostly the CPU working not so much the harddrive.

Thirdly, is there any way how to generate some output of what was copied, skipped, errors... and save it to a logfile?

$Source = 'D:\ZZZ_Phone_test'
$Dest = 'D:\ZZZ_Phone_test\1\1\BackUp'
$Skip = 'D:\ZZZ_Phone_test\4\.thumbnails'

Get-ChildItem $Source -Directory -Recurse | ? FullName -ne $Dest | ? FullName -ne $Skip | get-ChildItem -File | Copy-Item -Exclude `
*.0,*.1,*.nomedia,*.thumbnail,*.chck,*.crypt12,*.tmp,*.db,*.crypt1,*.ini,*.pdrproj,*.pkpass,*.dat,*.enc,*.lck,*.xml,*.json,*.LOCK,*.443,*.preference `
-Destination $Dest

.

EDIT: the following works but it will only exclude files in directories whose names end with "thumbnails" or "BackUp". If there are any directories with files inside of "thumbnails" folder they will all be processed. I'd like to define the folders to be excluded the way that even if there are subdirectories with files in a directory defined in $Skip they would not be processed.

$Source = 'D:\ZZZ_Phone_test'
$Dest   = 'D:\ZZZ_Phone_test\1\1\BackUp'
$Skip   = '*thumbnails', '*BackUp'


(Get-ChildItem -Path $Source -Directory -Recurse -Exclude $Skip).FullName |
 get-ChildItem -File |
 Copy-Item -WhatIf -Exclude `
*.0,*.1,*.nomedia,*.thumbnail,*.chck,*.crypt12,*.tmp,*.db,*.crypt1,*.ini,*.pdrproj,*.pkpass,*.dat,*.enc,*.lck,*.xml,*.json,*.LOCK,*.443,*.preference `
-Destination $Dest

Answer 1

For complex filtering needs, divide et impera is often useful an approach. That is, simplify the problem in multiple steps instead of trying to write an one-liner.

Let's take a directory listing of all the files and exclude the destination directory $dest. As % (shorthand for Where-Object) parameter -notmatch expects a regular expression, the $dest path is escaped with [regex]::Escape. This needs to be done, as backslash \ is a reserved character in regular expressions. One could write the path in escaped form in the first hand, like c:\\my\\path\\to\\somewhere, but Escape does all the work needed.

Get-ChildItem $source\* -Recurse -File | ? { $_.psparentpath -notmatch [regex]::escape($dest) }

Now that we have all the files except destination, start pruning the list. Since there are lots of file extensions, let's put those on an array. Loop through the array, and remove each match from the $files array.

$excluded = @("*.0", "*.1", "*.nomedia", "*.thumbnail", "*.chck", "*.crypt12", "*.tmp", "*.db",
   "*.crypt1", "*.ini", "*.pdrproj", "*.pkpass", "*.dat", "*.enc", "*.lck", "*.xml", "*.json", 
   "*.LOCK", "*.443", "*.preference")

foreach($ex in $excluded) {
    $files = $files | ? {$_.extension -notlike $ex}
}

To remove the $skip, filter the collection again:

$files = $files | ? {$_.DirectoryName -ne $skip)

At this point, all you have is an array that contains files that are to be copied into $dest. Before copying, use -WhatIf switch to see what Copy-Item would do to be sure the copy works as intended:

$files | % { Copy-Item -WhatIf $_ $dest }

To wrap up a complete example,

$source = "c:\temp\phonetest"
$dest = "c:\temp\phonetest\1\1\BackUp"
$Skip = "c:\temp\phonetest\skipme"
# Get list of all the files
Get-ChildItem $source\* -Recurse | ? { $_.psparentpath -notmatch [regex]::escape($dest) }

# Filter by extension
$excluded = @("*.0", "*.1", "*.nomedia", "*.thumbnail", "*.chck", "*.crypt12", "*.tmp", "*.db", "*.crypt1", "*.ini", "*.pdrproj", "*.pkpass", "*.dat", "*.enc", "*.lck", "*.xml", "*.json", "*.LOCK", "*.443", "*.preference")

foreach($ex in $excluded) {
$files = $files | ? {$_.extension -notlike $ex}
}

# Skip specific dir    
$files = $files | ? {$_.DirectoryName -ne $skip)

# See what would be copied
$files | % { Copy-Item -WhatIf $_ $dest }

Answer 2

Try this and modify as you wish for that file exclusion section...

$Source = 'D:\Temp'
$Dest   = 'D:\Destination'
$Skip   = '*est', '*here'

<#
Always build you code one use case at a time to ensure you are getting what 
you'd expect before moving ot the next.
#>

# Get all directories off a give path
(Get-ChildItem -Path $Source -Directory -Recurse).FullName | 
Select-Object -First 5
# Results
<#
D:\Temp\AddressFiles
D:\Temp\ChildFolder
D:\Temp\est
D:\Temp\here
D:\Temp\hold
#>

# Exclude named directories
(Get-ChildItem -Path $Source -Directory -Recurse -Exclude $Skip).FullName | 
Select-Object -First 5
# Results
<#
D:\Temp\AddressFiles
D:\Temp\ChildFolder
D:\Temp\ChildFolder\New folder
D:\Temp\ChildFolder\temp
D:\Temp\hold

#>

# Or include only what you want
(Get-ChildItem -Path $Source -Directory -Recurse -Include $Skip).FullName | 
Select-Object -First 5
# Results
<#
D:\Temp\ChildFolder\New folder\est
D:\Temp\est
D:\Temp\here
#>

# Loop directories and process files, trap errors
(Get-ChildItem -Path $Source -Directory -Recurse -Exclude $Skip).FullName | 
Select-Object -First 5 | 
ForEach {
    Try
    {
        "Processing $PSItem"
        $CopyItemSplat = @{
            Path        = (Get-ChildItem -Path $PSItem -ErrorAction Stop).FullName 
            Destination = $Dest 
            Verbose     = $true
            WhatIf      = $true
        }
    }
    Catch
    {
        Write-Warning -Message 'An error was encountered.'
        $PSitem.Exception.Message
    }
}
# Results
<#
Processing D:\Temp\AddressFiles
Processing D:\Temp\ChildFolder
Processing D:\Temp\ChildFolder\New folder
Processing D:\Temp\ChildFolder\temp
WARNING: An error was encountered.
The property 'FullName' cannot be found on this object. Verify that the property exists.
Processing D:\Temp\hold
WARNING: An error was encountered.
The property 'FullName' cannot be found on this object. Verify that the property exists.
#>