Using Powershell and Invoke-WebRequest to upload 2.5 GB to a web server

Question

I am a networking engineer and do not have any background on scripting or powershell however i started to deal with it lately as i need to create couple of scripts. I work for VMware and want to create a script that will automate the upgrade of the NSX environment and the first step to do so is to upload the upgrade bundle to the NSX Manager (which is a web-server at the end). From the API Guide of the NSX (https://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_api.pdf) you should use the below API to upload the upgrade bundle POST https://NSX-Manager-IP-Address/api/1.0/appliance-management/upgrade/uploadbundle/NSX So i am trying to create a script that will first ask you the required version to upgrade to and based on that i will put the file path in a variable and then use invoke-webrequest with the parameter -Infile to upload the upgrade bundle.

However when i do so the upload proccess starts but after couple of min i always get the following error (Invoke-WebRequest : The underlying connection was closed: An unexpected error occurred on a send.)

After some search i understood that the invoke-webrequest will first buffer the file which is 2.5GB and then upload it and possibly this is why the connection is closed as the NSX Manager did not receive anything yet.

Also i understood that i can disable the buffering using the below (as an example): $webRequest.AllowWriteStreamBuffering=$false $webRequest.SendChunked=$true

However i do not know how to add this to the script.

Below is the full script i have at the moment: PS: the powershell version i am using is 5

[CmdletBinding()]
$NSXUsername = "admin"
$NSXPassword = "VMware1!"
$uriP = "https://HQ-NSX-01a.nsx.gss"


# Start time.
$startclock = (Get-Date)
Write-Host -BackgroundColor:Black -ForegroundColor:Green "Hello"
Write-Host -BackgroundColor:Black -ForegroundColor:Green "This script will help you to automate a full NSX environment UPgrade"
Write-Host -BackgroundColor:Black -ForegroundColor:Green "FULL NSX Tier UPgrade for Single-VC is starting, This UPgrade proccess will take an average of 40 min
========================================================================================
                                 "




# Create NSX authorization string and store in $head and used in API Calls
# $nsxcreds = New-Object System.Management.Automation.PSCredential $NSXUsername,$NSXPassword
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($NSXUsername + ":" + $NSXPassword))
$head = @{"Authorization"="Basic $auth"}

# Allow untrusted SSL certs else will error out
    add-type @"
        using System.Net;
        using System.Security.Cryptography.X509Certificates;
        public class TrustAllCertsPolicy : ICertificatePolicy {
            public bool CheckValidationResult(
                ServicePoint srvPoint, X509Certificate certificate,
                WebRequest request, int certificateProblem) {
                return true;
            }
        }
"@
    [System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

##################################################
# Choosing the required NSX Version to be deployed
##################################################

[int]$menuoptions = 0
while ( $menuoptions -lt 1 -or $menuoptions -gt 12 ) {
Write-host -BackgroundColor:Black -ForegroundColor:Red "    Please choose the required version of NSX to be deployed (Accepted inputs are a number from 1 to 12)                    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     1.NSX-6.2.0 "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     2.NSX-6.2.1 "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     3.NSX-6.2.1a    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     4.NSX-6.2.2 "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     5.NSX-6.2.2a    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     6.NSX-6.2.2b    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     7.NSX-6.2.3 "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     8.NSX-6.2.3a    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     9.NSX-6.2.3b    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     10.NSX-6.2.4    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     11.NSX-6.2.5    "
Write-host -BackgroundColor:Black -ForegroundColor:Yellow "     12.NSX-6.3.6    "
[int]$menuoptions = read-host
if ($menuoptions -lt 1 -or $menuoptions -gt 12) {Write-host -BackgroundColor:Black -ForegroundColor:Red "       Invalid Input Detected, Please choose a valid input"}
}
Switch( $menuoptions ) {
1{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.0-2986609.tar.gz"}
2{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.1-3300239.tar.gz"}
3{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.1a-3496286.tar.gz"}
4{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.2-3604087.tar.gz"}
5{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.2a-3638734.tar.gz"}
6{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.2b-3755950.tar.gz"}
7{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.3-3979471.tar.gz"}
8{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.3a-4167369.tar.gz"}
9{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.3b-4287432.tar.gz"}
10{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.4-4292526.tar.gz"}
11{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.5-4818372.tar.gz"}
12{$nsxpath = "D:\NSX-6.2.x-UPGrade-Bundles\VMware-NSX-Manager-upgrade-bundle-6.2.6-4977495.tar.gz"}
}

Switch( $menuoptions ) {
1{$nsxversion = "NSX-6.2.0"}
2{$nsxversion = "NSX-6.2.1"}
3{$nsxversion = "NSX-6.2.1a"}
4{$nsxversion = "NSX-6.2.2"}
5{$nsxversion = "NSX-6.2.2a"}
6{$nsxversion = "NSX-6.2.2b"}
7{$nsxversion = "NSX-6.2.3"}
8{$nsxversion = "NSX-6.2.3a"}
9{$nsxversion = "NSX-6.2.3b"}
10{$nsxversion = "NSX-6.2.4"}
11{$nsxversion = "NSX-6.2.5"}
12{$nsxversion = "NSX-6.3.6"}
}


#===========================================================================================================================================    
#===========================================================================================================================================    


#########################
# Upgrading NSX Manager
#########################

    Write-Host -BackgroundColor:Black -ForegroundColor:Green "1. Deploying NSX Manager with version $nsxversion"

    Write-Host -BackgroundColor:Black -ForegroundColor:Yellow "     UPloading the required upgrade bundel to NSX Manager"


$r = Invoke-WebRequest -Uri "$uriP/api/1.0/appliance-management/upgrade/uploadbundle/NSX" -Method:Post -Headers $head -ContentType "application/xml" -InFile $nsxpath -TimeoutSec 66000 -DisableKeepAlive 

Any idea will be very helpful and very appreciated

Thank you for your help.

Answer 1

I think this kind of question has been asked before here: Upload BIG files via HTTP

It looks like the suggested solution is to use the [System.Net.HttpWebRequest] .NET class.

I haven't tested this but I think you can replace your $r = Invoke-WebRequest with something along these lines:

$webRequest = [System.Net.HttpWebRequest]::Create("$uriP/api/1.0/appliance-management/upgrade/uploadbundle/NSX")
$webRequest.Timeout = 66000
$webRequest.Method = "POST"
$webRequest.ContentType = "application/xml"
$webRequest.AllowWriteStreamBuffering=$false
$webRequest.SendChunked=$true
$webRequest.Credentials = $auth
$webRequest.Headers["Authorization"] = "Basic $auth"

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($nsxpath)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

$responseStream = $webRequest.getresponse()

$FileStream.Close()
$requestStream.Close()
$responseStream.Close()

$responseStream
$responseStream.GetResponseHeader("Content-Length") 
$responseStream.StatusCode