CODEWITHSUNDEEP
windowsbatch-fileamazon-s3cmdaws-cli
ThomasKa
ThomasKa

Reputation: 35

How do I avoid repeatedly starting cmd.exe in this batch file to run AWS CLI commands?

By following various examples, I've managed to cobble together a working batch file that uses the AWS CLI to extract file listings from AWS S3 buckets and append a URL to the start of them.

I know you can do something similar in programs like 'S3 Browser', but I needed a "one-click solution". The result is below; when I run it, it successfully extracts listings from 4 sub-folders and saves them to 4 different text files.

It definitely does what I want, but I think it's a pretty clumsy effort. For example, I'm starting, and then closing, cmd.exe on every loop to send commands.

I tried putting it outside the loop; it seems like it should be easy to do but, as a raw beginner to batch files, I can't get it to work.

Any ideas?

Here's the code:

@echo off 

REM ----------- SET VARIABLES -------------------
REM Set AWS Base URL
REM ----------------
set BaseURL=https://s3-eu-west-1.amazonaws.com

REM SET AWS Bucket Name
REM -------------------
set Bucket=XXXXXX

REM SET Bucket Sub-Folder
REM ---------------------
set sub-folder-1=XXXXXX

REM SET Bucket Sub-Folder
REM ---------------------
set sub-folder-2=XXXXXX

REM SET Bucket Sub-Folders
REM ----------------------
set sub-folder-3=MP4 MP3 PDF Thumbnails

REM Set AWS file download location
set DownloadAWSFileTo=C:\Desktop

REM ------------------------------------------------

REM Loop through all sub-folders
    For %%a in (%sub-folder-3%) do (
        REM Start AWS CLI, download/save directory list to desktop, and close CMD window
        start cmd.exe /k "aws s3 ls s3://%Bucket%/%sub-folder-1%/%sub-folder-2%/%%a/ > %DownloadAWSFileTo%\filelist.txt" ^& exit

    REM Pause for 3 seconds to make sure CMD is finished
        timeout /t 3

    REM Remove everything from AWS directory list EXCEPT file names
        for /f "tokens=4 delims= " %%i in (filelist.txt) DO ( echo %%i >> CLEANfilelist.txt)

    REM Append URL structure to beginning of each line and make new list
        for /F "delims=" %%j in (CLEANfilelist.txt) do echo.%BaseURL%/%Bucket%/%sub-folder-1%/%sub-folder-2%/%%a/%%j >> %%a-URLs-%sub-folder-2%-AWS.txt

    REM Delete unwanted text files
        del filelist.txt
        del CLEANfilelist.txt
   )

Upvotes: 1

Views: 779

Answers (1)

ThomasKa
ThomasKa

Reputation: 35

Just removing cmd.exe did the trick...

REM Loop through all sub-folders
For %%a in (%sub-folder-3%) do (
    REM AWS CLI command to get directory file list and save to desktop
        aws s3 ls s3://%Bucket%/%sub-folder-1%/%sub-folder-2%/%%a/ > %DownloadAWSFileTo%\filelist.txt

    REM Remove everything from AWS directory list EXCEPT file names
        for /f "tokens=4 delims= " %%i in (filelist.txt) do (echo %%i >> CLEANfilelist.txt)

    REM Append URL structure to beginning of each line and make new list
        for /F "delims=" %%j in (CLEANfilelist.txt) do echo.%BaseURL%/%Bucket%/%sub-folder-1%/%sub-folder-2%/%%a/%%j >> %%a-URLs-%sub-folder-2%-AWS.txt

    REM Delete unwanted text files
        del filelist.txt
        del CLEANfilelist.txt
)

REM Pause to finish final file cleanup
timeout /t 3

Upvotes: 1

Related Questions