CODEWITHSUNDEEP
powershellconcurrencynumberslimitjobs
user3360660
user3360660

Reputation: 1

Limit number of concurrent powershell jobs

I know this question has been asked before and i have seen a couple of possible solutions, but i can't seem to be able to make any of them work for my script. My script is not a finished product, needs some cleaning up, but it definitely works at the moment.

The basic function of the script is that it creates snapshots of virtual machines, and then clones them for backup purposes. Everything works as expected, but the problem is it brings my vcenter to it's knees because it is flooding it with too many jobs at once. I was wanting to limit the amount of jobs that run at once to maybe 3-4 at the same time. Can anyone give me a hand? Thanks in advance... Here is my code:

Add-PSSnapin VMware.VimAutomation.Core

#Connect to vCenter
Connect-VIServer vcenter.mydomain.com

#Define variables
$vmhost = get-vmhost -name 10.1.1.10

#Export list of VM's
get-vmhost -name $vmhost | get-vm | select Name, BackupDS | Export-Csv -NoTypeInformation test.csv

#Edit CSV to populate BackupDS Column
(Import-Csv test.csv) | % { $_.BackupDS = '[NAS1]' +$_.BackupDS; $_ } | Export-Csv test.csv -NoTypeInformation

#Edit CSV file to format required by script
(gc test.csv) | foreach-object{$_ -replace 'Name','MasterVM'} | set-content test.csv -force

#Import Backup CSV
$backupinfo =  Import-Csv test.csv



$scriptblock = {
Param($customer)

    Add-PSSnapin VMware.VimAutomation.Core

    #Connect to vCenter
    Connect-VIServer vcenter.mydomain.com

    #Define variables
    $vmhost = get-vmhost -name 10.1.1.10

    #Set Date format for clone names
    $date = Get-Date -Format "MMddyyyy"

    #Set Date format for emails
    $time = (Get-Date -f "HH:MM")

    $vm = Get-VM $customer.MasterVM

    #Send Start Email
    #C:\scripts\backupstartedemail.ps1

    # Create new snapshot for clone
    $cloneSnap = $vm | New-Snapshot -Name "Clone Snapshot"

    # Get managed object view
    $vmView = $vm | Get-View

    # Get folder managed object reference
    $cloneFolder = $vmView.parent

    # Build clone specification
    $cloneSpec = new-object Vmware.Vim.VirtualMachineCloneSpec
    $cloneSpec.Snapshot = $vmView.Snapshot.CurrentSnapshot

    # Make linked disk specification
    $cloneSpec.Location = new-object Vmware.Vim.VirtualMachineRelocateSpec
    $cloneSpec.Location.Datastore = (Get-Datastore -Name $customer.BackupDS | Get-    View).MoRef
    $cloneSpec.Location.Transform =      [Vmware.Vim.VirtualMachineRelocateTransformation]::sparse

    $cloneName = "$vm-$date"

    # Create clone
    $vmView.CloneVM( $cloneFolder, $cloneName, $cloneSpec )

    # Write newly created VM to stdout as confirmation
    Get-VM $cloneName

    # Remove Snapshot created for clone
    Get-Snapshot -VM (Get-VM -Name $customer.MasterVM) -Name $cloneSnap | Remove-    Snapshot -confirm:$False

    # Remove clones machine from inventory
    remove-vm -vm $cloneName -confirm:$false

    #Send Complete Email
    #C:\scripts\backupcompletedemail.ps1
}

$backupinfo | % {Start-Job -Scriptblock $scriptblock -ArgumentList $_ | Out-Null}

Get-Job | Wait-Job | Receive-Job

I've tried throwing the following code at the end of the script with no luck:

$backupinfo | % {Start-Job -Scriptblock $scriptblock -ArgumentList $_ | Out-Null}

While((Get-Job -State 'Running').Count -ge 3)
    {
        Start-Sleep -Milliseconds 10
    }

Update**

I tried using the answer provided here, Run N parallel jobs in powershell. 

Add-PSSnapin VMware.VimAutomation.Core

#Connect to vCenter
Connect-VIServer vcenter.mydomain.com

#Define variables
$vmhost = get-vmhost -name 10.1.1.10

#Export list of VM's
get-vmhost -name $vmhost | get-vm | select Name, BackupDS | Export-Csv -NoTypeInformation     test.csv

#Edit CSV to populate BackupDS Column
(Import-Csv test.csv) | % { $_.BackupDS = '[NAS1]' +$_.BackupDS; $_ } | Export-Csv test.csv     -NoTypeInformation

#Edit CSV file to format required by script
(gc test.csv) | foreach-object{$_ -replace 'Name','MasterVM'} | set-content test.csv -force

#Import Backup CSV
$backupinfo =  Import-Csv test.csv

#Set Date format for clone names
$date = Get-Date -Format "MMddyyyy"

#Set Date format for emails
$time = (Get-Date -f "HH:MM")

foreach ($customer in $backupinfo) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le 8) {
        Start-Job {
             $vm = Get-VM $customer.MasterVM

    #Send Start Email
    #C:\scripts\backupstartedemail.ps1

    # Create new snapshot for clone
    $cloneSnap = $vm | New-Snapshot -Name "Clone Snapshot"

    # Get managed object view
    $vmView = $vm | Get-View

    # Get folder managed object reference
    $cloneFolder = $vmView.parent

    # Build clone specification
    $cloneSpec = new-object Vmware.Vim.VirtualMachineCloneSpec
    $cloneSpec.Snapshot = $vmView.Snapshot.CurrentSnapshot

    # Make linked disk specification
    $cloneSpec.Location = new-object Vmware.Vim.VirtualMachineRelocateSpec
    $cloneSpec.Location.Datastore = (Get-Datastore -Name $customer.BackupDS | Get-    View).MoRef
    $cloneSpec.Location.Transform =      [Vmware.Vim.VirtualMachineRelocateTransformation]::sparse

    $cloneName = "$vm-$date"

    # Create clone
    #$vmView.CloneVM( $cloneFolder, $cloneName, $cloneSpec )

    # Write newly created VM to stdout as confirmation
    #Get-VM $cloneName

    # Remove Snapshot created for clone
    Get-Snapshot -VM (Get-VM -Name $customer.MasterVM) -Name $cloneSnap | Remove-Snapshot -     confirm:$False

    # Remove clones machine from inventory
    #remove-vm -vm $cloneName -confirm:$false

    #Send Complete Email
    #C:\scripts\backupcompletedemail.ps1
        }
    } else {
         $running | Wait-Job
    }
    Get-Job | Receive-Job
}

When i run my script all it does is return this for each virtual machine, and never does any of the tasks (snapshots, etc.)

HasMoreData   : True
StatusMessage :
Location      : localhost
Command       :
                     $vm = Get-VM $customer.MasterVM
                     ..................
                     rest of my code
                     ..................
JobStateInfo  : Running
Finished      : System.Threading.ManualResetEvent
InstanceId    : 81ac8e67-0267-4d11-998b-0e8cfa95292b
Id            : 40
Name          : Job40
ChildJobs     : {Job41}
PSBeginTime   : 2/27/2014 12:41:19 PM
PSEndTime     :
PSJobTypeName : BackgroundJob
Output        : {}
Error         : {}
Progress      : {}
Verbose       : {}
Debug         : {}

Any guidance you can provide?

Upvotes: 0

Views: 2186

Answers (1)

Keith Hill
Keith Hill

Reputation: 201952

Consider using a workflow. It has a parallel foreach that should intelligently throttle the number of concurrent executions e.g.:

workflow New-Clone([string[]]$Customer) {
    InlineScript { Import-Module modules }
    foreach -parallel($cust in $Customer) {
        InlineScript {
            ...
            $vm = Get-VM $using:cust.MasterVM
            ...
        } 
    }
}

This does require PowerShell V3 or higher for the workflow support.

Upvotes: 1

Related Questions