CODEWITHSUNDEEP
powershellbackground-processjobs
Daniel
Daniel

Reputation: 590

Timeout Invoke-Command Background Child Jobs

I'm creating a controller script that will run audit scripts contained in separate .ps1 files. I'm using Invoke-Command to kick off jobs on target servers. I know that this will create child jobs for each target computer. How can I timeout individual child jobs if they run too long (ex: 2 minutes)? I know there's Wait-Job -Timeout <seconds>, but that won't really work for my scenario since I want to also print out progress. Below is what I've tried so far, however the timeout doesn't appear to function.

Any ideas/suggestions? Thanks!

    Write-Host "Queueing job... " -NoNewline
    Invoke-Command -FilePath $JobFilePath -ComputerName $Targets -AsJob -ThrottleLimit 16 -JobName "AuditJob" -ErrorAction SilentlyContinue | Out-Null
    Invoke-Command -ScriptBlock {Get-Job | Wait-job -Timeout 15} -ComputerName $Targets -AsJob -ThrottleLimit 16 -ErrorAction SilentlyContinue | Out-Null
    Write-Host "Done" -ForegroundColor Green

    $origpos = $host.UI.RawUI.CursorPosition

    [Console]::CursorVisible = $false
    
    $Completed      = 0
    $NotStarted     = 0
    $Running        = 0
    $Stopped        = 0
    $Failed         = 0
    $Blocked        = 0
    $Total          = 0
    $JobsComplete   = 0
    
    $CompleteStrLength = 0

    do{
        $Job            = Get-Job -Name "AuditJob" -IncludeChildJob -ErrorAction Stop | Where-Object {$_.Name -ne "AuditJob"}
        $Completed      = ($Job | Where-Object {$_.State -eq "Completed"}).Count
        $NotStarted     = ($Job | Where-Object {$_.State -eq "NotStarted"}).Count
        $Running        = ($Job | Where-Object {$_.State -eq "Running"}).Count
        $Stopped        = ($Job | Where-Object {$_.State -eq "Stopped"}).Count
        $Failed         = ($Job | Where-Object {$_.State -eq "Failed"}).Count
        $Blocked        = ($Job | Where-Object {$_.State -eq "Blocked"}).Count
    
        $Total = $Completed + $NotStarted + $Running +$Stopped + $Failed + $Blocked
        $JobsComplete = $Total - $NotStarted - $Running
        $PercentComplete = $JobsComplete/$Total

        $host.UI.RawUI.CursorPosition = $origpos
        
        Write-Host "[In Progress: $Running || Completed: $Completed || Not Started: $NotStarted || Failed: $($Stopped + $Failed + $Blocked)]"
        $CurrentPos = $host.UI.RawUI.CursorPosition
        Write-Host (' ' * $CompleteStrLength)
        $host.UI.RawUI.CursorPosition = $CurrentPos
        
        $CompleteStr = "$([Math]::Ceiling($PercentComplete * 100))% complete"
        $CompleteStrLength = $CompleteStr.Length
        Write-Host $CompleteStr
        Start-Sleep -Milliseconds 250 #helps with excessive flashing

    }while ($JobsComplete -ne $Targets.Count)

Upvotes: 1

Views: 203

Answers (1)

kconsiglio
kconsiglio

Reputation: 665

So, I think one issue is with your second Invoke-Command. From what I understand the job is being created on your local computer, so that command won't really be doing anything since you are trying to wait for a job that won't exist on the target computer. Using Wait-Job might work locally but getting it to work while your interface is being updated at the same time could be tricky since it pauses the script until the set timeout is reached.

To work around this I created a foreach loop inside your loop to go through the job objects. For each job object, I got start time of the job using the PSBeginTime property and got the elapsed time by subtracting that from the current time using Get-date. If that elapsed is over your set timeout you can use Stop-Job to stop the job.

It does stop the job on the remote machine from continuing and will report on your output as failed. Could require more testing if your remote job is launching an .exe or something as I only tested with a powershell script.

$JobFilePath = ""
$targets = ""
# change as required
$timeOutSeconds = 2
Write-Host "Queueing job... " -NoNewline

Write-Host "Queueing job... " -NoNewline
    Invoke-Command -FilePath $JobFilePath -ComputerName $targets -AsJob -ThrottleLimit 16 -JobName "AuditJob" -ErrorAction SilentlyContinue | Out-Null
    # remove line
    # Invoke-Command -ScriptBlock {Get-Job | Wait-job -Timeout 15} -ComputerName $Targets -AsJob -ThrottleLimit 16 -ErrorAction SilentlyContinue | Out-Null
    Write-Host "Done" -ForegroundColor Green

    $origpos = $host.UI.RawUI.CursorPosition

    [Console]::CursorVisible = $false
    
    $Completed      = 0
    $NotStarted     = 0
    $Running        = 0
    $Stopped        = 0
    $Failed         = 0
    $Blocked        = 0
    $Total          = 0
    $JobsComplete   = 0
    
    $CompleteStrLength = 0

    do{
        $Job            = Get-Job -Name "AuditJob" -IncludeChildJob -ErrorAction Stop | Where-Object {$_.Name -ne "AuditJob"}
        $Completed      = ($Job | Where-Object {$_.State -eq "Completed"}).Count
        $NotStarted     = ($Job | Where-Object {$_.State -eq "NotStarted"}).Count
        $Running        = ($Job | Where-Object {$_.State -eq "Running"}).Count
        $Stopped        = ($Job | Where-Object {$_.State -eq "Stopped"}).Count
        $Failed         = ($Job | Where-Object {$_.State -eq "Failed"}).Count
        $Blocked        = ($Job | Where-Object {$_.State -eq "Blocked"}).Count

        foreach ($checkJOb in $Job){   
            
        # if job is not started yet time will be null and will error out
        if ($null -ne $checkJOb.PSBeginTime){
                
            # calculate elapsed time in seconds.
            $totalSecondsElapsed = ((Get-Date) - ($checkJOb.PSBeginTime)).TotalSeconds

            # check if time elapsed has passed timeout threshold
            if ($totalSecondsElapsed -ge $timeOutSeconds){               

                # stop the current job
                Stop-Job  -id  $checkJOb.id             

            }
        }     
            
            
        }
    
        $Total = $Completed + $NotStarted + $Running +$Stopped + $Failed + $Blocked
        $JobsComplete = $Total - $NotStarted - $Running
        $PercentComplete = $JobsComplete/$Total

        $host.UI.RawUI.CursorPosition = $origpos
        
        Write-Host "[In Progress: $Running || Completed: $Completed || Not Started: $NotStarted || Failed: $($Stopped + $Failed + $Blocked)]"
        $CurrentPos = $host.UI.RawUI.CursorPosition
        Write-Host (' ' * $CompleteStrLength)
        $host.UI.RawUI.CursorPosition = $CurrentPos
        
        $CompleteStr = "$([Math]::Ceiling($PercentComplete * 100))% complete"
        $CompleteStrLength = $CompleteStr.Length
        Write-Host $CompleteStr
        Start-Sleep -Milliseconds 250 #helps with excessive flashing

    }while ($JobsComplete -ne $Targets.Count)

Upvotes: 1

Related Questions