CODEWITHSUNDEEP
powershellrandom
Denis
Denis

Reputation: 53

PowerShell random file generator too slow

Trying to generate random files in Windows with this PowerShell script:

param([string]$path = "C:\temp", [int]$size = "1024", [long]$number= "1000")
Write-Host "Path: $path"
Write-Host "Size of file: $size"
Write-Host "Number of files: $number"

Write-Host "Started at:"
Get-Date -Format HH:mm:ss

md -Force $path |out-null

$script:StartTime = $(Get-Date)
for($i=0; $i -le $number - 1 ; $i++){
    [Byte[]]$out=@(); 0..($size-1) | %{$out += Get-Random -Minimum 0 -Maximum 255};
    [System.IO.File]::WriteAll("$path\$i.bin",$out)
    if (-Not ($i % 100) -and ($i -ne 0 )) {
        Write-Host "$i were created"
        $elapsedTime = $(Get-Date) - $script:StartTime
        $script:StartTime = $(Get-Date)
        Write-Host "$elapsedTime"
    }
}

When I want to increase filesize from 1KB to 1MB, it take a long time for even one file, but I need thousands of those. Is there way to solve this bottleneck?

Upvotes: 1

Views: 2758

Answers (2)

Marc Kellerman
Marc Kellerman

Reputation: 466

Function New-RandomFile {
    Param(
        $Path = '.', 
        $FileSize = 1kb, 
        $FileName = [guid]::NewGuid().Guid + '.txt'
        ) 
    (1..($FileSize/128)).foreach({-join ([guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid -Replace "-").SubString(1, 126) }) | set-content "$Path\$FileName"
}

This took 491 milliseconds to generate a 1mb file. Running:

New-RandomFile -FileSize 1mb

UPDATE:

I've updated my function to use a ScriptBlock, so you can replace the 'NewGuid()' method with anything you want.

In this scenario, I make 1kb chunks, since I know I'm never creating smaller files. This improved the speed of my function drastically!

Set-Content forces a NewLine at the end, which is why you need to remove 2 Characters each time you write to file. I've replaced it with [io.file]::WriteAllText() instead.

Function New-RandomFile_1kChunks {
    Param(
        $Path = (Resolve-Path '.').Path, 
        $FileSize = 1kb, 
        $FileName = [guid]::NewGuid().Guid + '.txt'
        ) 

    $Chunk = { [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid -Replace "-" }

    $Chunks = [math]::Ceiling($FileSize/1kb)

    [io.file]::WriteAllText("$Path\$FileName","$(-Join (1..($Chunks)).foreach({ $Chunk.Invoke() }))")

    Write-Warning "New-RandomFile: $Path\$FileName"

}

If you dont care that all chunks are random, you can simply Invoke() the generation of the 1kb chunk once.. this improves the speed drastically, but won't make the entire file random.

Function New-RandomFile_Fast {
    Param(
        $Path = (Resolve-Path '.').Path, 
        $FileSize = 1kb, 
        $FileName = [guid]::NewGuid().Guid + '.txt'
        ) 

    $Chunk = { [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid +
               [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid + [guid]::NewGuid().Guid -Replace "-" }
    $Chunks = [math]::Ceiling($FileSize/1kb)
    $ChunkString = $Chunk.Invoke()

    [io.file]::WriteAllText("$Path\$FileName","$(-Join (1..($Chunks)).foreach({ $ChunkString }))")

    Write-Warning "New-RandomFile: $Path\$FileName"

}

Measure-Command all these changes to generate a 10mb file:

Executing New-RandomFile: 35.7688241 seconds.

Executing New-RandomFile_1kChunks: 25.1463777 seconds.

Executing New-RandomFile_Fast: 1.1626236 seconds.

Upvotes: 1

Denis
Denis

Reputation: 53

Solved with dd for windows.

#default values

param([string]$path = "C:\temp", [int]$size = "1048576", [long]$number= "10000")
Write-Host "Path: $path"
Write-Host "Size of file: $size"
Write-Host "Number of files: $number"

##dd.exe should be in same dir
$path_to_dd = Get-Location 

Write-Host "Started at:"
Get-Date -Format HH:mm:ss

md -Force $path |out-null

$script:StartTime = $(Get-Date)

for($i=0; $i -le $number - 1 ; $i++){
    #genarate files
    (& $path_to_dd\dd.exe if=/dev/random of=$path\$i.bin bs=$size count=1) 2>&1 | Out-Null
    #periodical report to stdout with timestamp every 1k files
    if (-Not ($i % 1000) -and ($i -ne 0 )) {
        Write-Host "$i were created"
        $elapsedTime = $(Get-Date) - $script:StartTime
        $script:StartTime = $(Get-Date)
        Write-Host "$elapsedTime"
    }
}

Write-Host "Finished at:"
Get-Date -Format HH:mm:ss

Upvotes: 0

Related Questions