user2917347
user2917347

Reputation: 99

Batch file to upload to S3

I needed to develop a batch file, which would send multiple small text files (4KB – 100KB) to a folder in an S3 bucket. Inspired by the bash script from http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash, I developed the batch script listed below. This script uses the following dependencies:

It works well; however, from time to time a file or multiple files will not upload and I’ll receive “HTTP/1.1 403 Forbidden / SignatureDoesNotMatch.” This seems to be a signature issue, possibly, when the signature is created. There does not seem to be a pattern to the behavior. For testing, I am using 8 text files to upload. I can run it multiple times resulting in all files uploading each time, and the next time the script will only upload 5, 6, or 7 of the 8. I am kind of at a loss to why this is happening. Any suggestions would be appreciated.

@echo off
setlocal enableDelayedExpansion

for /f %%f in ('dir /b *.txt') do (

REM Set date/time variables
for /f "tokens=1-4 delims=/ " %%a in ('date /t') do set dow=%%a&&set day=%%c&& set month=%%b&&set year=%%d
for /f "tokens=1-4 delims=:,. " %%h in ('echo %time%') do set hour=%%h&set min=%%i&set sec1=%%j&set sec2=%%k

REM Obtain three letter month value for DateValue HTTP Header
IF !month! == 01 set mname=Jan
IF !month! == 02 set mname=Feb
IF !month! == 03 set mname=Mar
IF !month! == 04 set mname=Apr
IF !month! == 05 set mname=May
IF !month! == 06 set mname=Jun
IF !month! == 07 set mname=Jul
IF !month! == 08 set mname=Aug
IF !month! == 09 set mname=Sep
IF !month! == 10 set mname=Oct
IF !month! == 11 set mname=Nov
IF !month! == 12 set mname=Dec

REM Set time as a 4 digit value if leading zero does not exist
for /f "tokens=1" %%u in ('echo %time%') do set t=%%u
IF "!hour:~1,1!"==":" set t=0!hour!

REM Obtain the ActiveBias value and convert to decimal 
for /f "tokens=3" %%a in ('reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation /v ActiveTimeBias ^| grep -i "ActiveTimeBias"') do set /a abias=%%a

REM Set the + or - sign variable to reflect the timezone offset
IF "!abias:~0,1!"=="-" (set si=+) ELSE (set si=-)
for /f "tokens=1 delims=-" %%t in ('echo !abias!') do set tzc=%%t

REM Calculate to obtain floating points (decimal values)
set /a tzd=100*!tzc!/60

REM Calculate the active bias to obtain the hour
set /a tze=!tzc!/60

REM Set the minutes based on the result of the floating point calculation
IF "!tzd!"=="0" (set en=00 && set si=)
IF "!tzd:~1!"=="00" (set en=00) ELSE IF "!tzd:~2!"=="00" (set en=00 && set tz=!tzd:~0,2!) 
IF "!tzd:~1!"=="50" (set en=30) ELSE IF "!tzd:~2!"=="50" (set en=30 && set tz=!tzd:~0,2!) 
IF "!tzd:~1!"=="75" (set en=45) ELSE IF "!tzd:~2!"=="75" (set en=45 && set tz=!tzd:~0,2!) 

REM Adding a 0 to the beginning of a single digit hour value
IF !tze! LSS 10 (set tz=0!tze!) 

REM Set the date/timestamp to meet required format
set dateValue=!dow!, !day! !mname! !year! !hour!:!min!:!sec1! !si!!tz!!en!

REM Preparing the HTTP header field
set file=%%f
set bucket=yourbucketname
set resource=/!bucket!/upload/!file!
set contentType=text/plain

REM You MUST have two returns after set NL=^
setlocal enableDelayedExpansion
set NL=^


set stringToSign=PUT!NL!!NL!!contentType!!NL!!dateValue!!NL!!resource!
<nul set /p ".=!stringToSign!" > put.tmp

set S3KEY="Your S3 Key Here"
set S3SECRET="Your S3 Secrect Key Here"

for /f "tokens=*" %%a in ('type put.tmp ^| openssl sha1 -hmac !S3SECRET! -binary ^| b64 -e') do set signature=%%a

REM Sending the data
curl -vvv --no-alpn --http2 -1 -S -X PUT -T "!file!" -H "Host: !bucket!.s3.amazonaws.com" -H "Date: !dateValue!" -H "Content-Type: !contentType!" -H "Authorization: AWS !S3KEY!:!signature!" "https://!bucket!.s3.amazonaws.com/upload/!file!"

REM Reset Variables
set "day="
set "month="
set "year="
set "retry="
set "dow="
set "hour="
set "min="
set "sec1="
set "sec2="
set "dateValue="
set "file="
set "mname="
set "bucket="
set "resource="
set "contentType="
set "stringToSign="
set "S3KEY="
set "S3SECRET="
set "t="

del put.tmp

)

Upvotes: 3

Views: 6177

Answers (1)

Martin Prikryl
Martin Prikryl

Reputation: 202272

Why don't you use a Windows command-line client with an explicit S3 support? Your batch file cannot do without 3rd party dependencies anyway.

Amazon has its own aws-cli.


If you need a more lightweight solution, use WinSCP:

winscp.com /log=s3.log /command ^
    "open s3://%S3KEY%:%S3SECRET%@s3.amazonaws.com/" ^
    "put *.txt /%BUCKET%/" ^
    "exit"

You need to URL-encode special characters in the credentials. WinSCP GUI can generate an S3 script template, like the one above, for you.

Alternatively, since WinSCP 5.19, you can use -username and -password switches, which do not need any encoding:

winscp.com /log=s3.log /command ^
    "open s3://s3.amazonaws.com/ -username=%S3KEY% -password=%S3SECRET%" ^
    "put *.txt /%BUCKET%/" ^
    "exit"

(I'm the author of WinSCP)

Upvotes: 5

Related Questions