CODEWITHSUNDEEP
performancerestpowershellloopsram
user1416876
user1416876

Reputation: 99

Improve performance in powershell file generation

I have written a basic script that gets system logs from a restful api, and writes it to a json file. It works, but im seeing huge RAM issues - i think its because the script is getting the whole payload of system log entries and storing it in memory before writing it to the file.

The do loop in the script below is for pagination - the api only returns a certain number of events at a time so it needs to be iterated through.

Can performance be improved here - I guess can the payload in memory get written to a file when it hits a certain size, and then it just loops over? Or is there anything else that can be done to reduce the system resources?

Thanks for your help!

#Name: getLogs.ps1
#Purpose: Script for exporting logs to a json file

#variables
$org = "<api url here>"
$token="<api token here>"
$filePath = "D:\Downloads\logs\test2.json"
#format the date and append 00:00:00
$fromTime = Get-Date -Format s
$fromTime = $fromTime.Substring(0,10) + "00%3A00%3A00Z"
#format the date and append 23:59.59
$toTime = Get-Date -Format s
$toTime = $fromTime.Substring(0,10) + "T23%3A59%3A59Z"

### Set $uri as the API URI for use in the loop
$uri = "$org/api/v1/logs?until=$toTime&limit=20&sortOrder=DESCENDING&q=&since=$fromTimeT00"

### Define $allLogsas empty array
$allLogs = @()

### Use a while loop and get all users from Okta API
DO
{
$webrequest = Invoke-WebRequest -Headers @{"Authorization" = "SSWS $token"} -Method Get -Uri $uri
$link = $webrequest.Headers.Link.Split("<").Split(">") 
$uri = $link[3]
$json = $webrequest | ConvertFrom-Json
$allLogs += $json
#slow it down to avoid rate limits
Start-Sleep -m 1001
} while ($webrequest.Headers.Link.EndsWith('rel="next"'))

# get results, switch to json, save

$allLogs | ConvertTo-Json | Out-File $filePath

Upvotes: 0

Views: 359

Answers (2)

postanote
postanote

Reputation: 16096

There is another post about performance and using the PoSH Invoke-* cmdlets. As noted in the previous post, there are just slow in general prior to PoSHv6. See this post for a that on the topic. Choppy File Download using Powershell on Azure VM

Upvotes: 1

Mark Wragg
Mark Wragg

Reputation: 23355

You could try changing:

$json = $webrequest | ConvertFrom-Json

To:

$webrequest | Out-File $filePath -Append

And then doing away with $alllogs entirely.

Upvotes: 1

Related Questions