CODEWITHSUNDEEP
c#asp.netamazon-s3streamingasp.net-core-2.0
Kyle Brown
Kyle Brown

Reputation: 39

Stream file upload from client directly to s3 through ASP.net-core2 app

Background: A user has a file to upload paired with some other multiform text data. This is POST'd to our ASP.net core 2 application which then has some logic built around the successful download of the said file.

I know I can generate a pre-signed S3 URL for the client to consume but we are looking to avoid that.

The problem I have is being unable to directly stream the file to S3. We can save the file locally and then stream that file to S3 (successfully) but we want to avoid this as we will have scaling issues. We want to minimize the caching of data to stream to S3 if possible, as well as not save any files locally to disc for upload.

I've tried multiple flavours of 

TransferUtility.UploadAsync(stream, ...)  
IAmazonS3.UploadPartAsync(...)  
IAmazonS3.UploadObjectFromStreamAsync(...)

https://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFileDotNet.html

and a few other ways which I can no longer remember.

This is one (if not the major) problem I've come across in most attempts https://github.com/aws/aws-sdk-net/issues/675

How can I use the HTTP stream provided to upload said data to S3?

This is how I'm handling my file stream in my resultscontroller:

[HttpPost("myupload")]
public async Task<IActionResult> TestBlobAsync()
{
    result = await this._internMethod(Request);
}

I'm testing my software using the following POSTMAN HTTP profile

POST /api/result/myupload HTTP/1.1
Host: localhost:62788
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Cache-Control: no-cache
Postman-Token: a17bef73-6015-c547-6f41-bc47274cb679

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="result"; filename="dummy-35MB.bin"
Content-Type: application/octet-stream


------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="value1"

2
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="value2"

8
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Please let me know if I can provide any other details.

Upvotes: 1

Views: 3370

Answers (1)

Kyle Brown
Kyle Brown

Reputation: 39

My solutuion was to change 

[HttpPost("myupload")]
public async Task<IActionResult> TestBlobAsync()
{
    result = await this._internMethod(Request);
}

to

[HttpPost("myupload")]
public async Task<IActionResult> TestBlobAsync(IFileForm file)
{
    ...
}

I am now able to use Stream stream = file.OpenReadStream() and pass it to TransferUtility.UploadAsync(stream, ...) and successfully get around the lack of content-length issues I was having leveraging the Request object manually.

Upvotes: 2

Related Questions