CODEWITHSUNDEEP
s3cmd
user2395365
user2395365

Reputation: 2149

Can s3cmd be used to download a file and upload to s3 without storing locally?

I'd like to do something like the following but it doesn't work:

wget http://www.blob.com/file | s3cmd put s3://mybucket/file

Is this possible?

Upvotes: 1

Views: 1504

Answers (2)

Chris
Chris

Reputation: 3466

To answer the question regarding s3cmd: No, it can not (currently) read from STDIN.

It does support multi-part-upload and also streams to STDIN, but apparently not the other way around.

Piping output from s3cmd works like this:

s3cmd get s3://my-bucket/some_key - | gpg -d | tar -g /dev/null -C / -xvj

Please be aware that there may be an issue with streaming gzip files: https://github.com/s3tools/s3cmd/issues/811

Upvotes: 0

Harshavardhana
Harshavardhana

Reputation: 1428

Cannot speak for s3cmd but its definitely possible.

You can use https://github.com/minio/mc . Minio Client aka mc is written in Golang, released under Apache License Version 2.

It implements mc pipe command for users to stream data directly to Amazon S3 from an incoming data on a pipe/os.stdin. mc pipe can also pipe to multiple destinations in parallel. Internally mc pipe streams the output and does multipart upload in parallel. 

$ mc pipe
NAME:
   mc pipe - Write contents of stdin to files. Pipe is the opposite of cat command.

$ mc cat
NAME:
   mc cat - Display contents of a file.

Example

#!/bin/bash

mc cat https://s3.amazonaws.com/mybucket/1.txt | mc pipe https://s3-us-west-2.amazonaws.com/mywestbucket/1.txt

Upvotes: 2

Related Questions