CODEWITHSUNDEEP
amazon-web-servicesgoamazon-s3file-upload
Milos
Milos

Reputation: 73

Can file just pass trough the server and to AWS S3?

I'm trying to upload large file to S3 but:

What I'm trying to accomplish is: as soon as file upload stars - pipe that file upload directly to S3 so that I don't have to wait for file to be uploaded to the server and then wait again for file to be uploaded to S3.

I've only started using GoLang and I mostly have no idea what I'm doing.

So far I'm using echo (https://echo.labstack.com).

file, err := c.FormFile("file")
if err != nil {
    return err
}

src, err := file.Open()
if err != nil {
    return err
}
defer src.Close()

If I understood correctly at this point i have io.Reader (src variable) and I don't have to wait for file to be uploaded to server, before I can use that reader to start uploading to S3? Basically while file is being sent to the server, at the same time server is sending the file to S3.

Or am I wrong? Is it better to use io.Pipe?

Upvotes: 1

Views: 526

Answers (2)

Kelsnare
Kelsnare

Reputation: 705

Actually you don't really "have" io.Reader. It is just an interface provided by the io package. io.Pipe, if you are interested, returns

(*io.PipeReader, *io.PipeWriter)

The io.PipeReader above is an actual implementation of the io.Reader interface - and hence can be used directly in golang aws sdk for s3 when uploading.

Here's how io.PipeReader works

Upvotes: 1

duyvh
duyvh

Reputation: 494

I'm assuming you want to upload a file from a client (web, mobile, desktop applications) to S3 without storing the file in your server.

A possible approach using official AWS SDK is answered here: Upload object to AWS S3 without creating a file using aws-sdk-go

Alternatively, if you want to skip your server entirely for uploading and let your client apps upload directly to your S3 bucket, you might want to take a look at S3 pre-signed upload URL.

Your server now plays the role of pre-signing the upload request, meaning it will authorize clients to upload the files to your S3 bucket. For large file, you will have to use multi-part upload and your server will need to pre-sign each part.

Upvotes: 2

Related Questions