Reputation: 1415
User Case: Customer can upload the file from the public REST api to our S3 bucket and then we can process the file using downstream services.
After doing some research I am able to find 3 ways to do it:
In first 2 cases user will send the binary file and we will upload the file to S3 after file validation.
In the 3rd method user have to hit 3 apis. First API to get the S3 pre-signed URL which will give access to the user to upload the file to S3. In second hit user will upload the file to that s3 pre-signed URL. After the user complete the upload he will send the request to process the file.
Do we have any security issues with step 3? As user can misuse the pre-signed URL with malicious file.
Which of these method is best according to industry practice?
Details of each approach:
1. Uploading using OCTET-STREAM file type
Pros:
Cons:
2. Upload the file using form-data request
User will upload the file with the API request by attaching it as multipart form.
Pros
3. Upload the file using the pre-signed URL
Cons
Upvotes: 2
Views: 7751
Reputation: 1415
As per my findings industry practice often favors using pre-signed URLs. Some pointers:
Scalability: Offloading the upload process to the client reduces server load. Security: Pre-signed URLs can be configured with specific permissions and expiration times, minimizing misuse risks. Flexibility: Supports large file uploads and can handle various file types and sizes.
To mitigate security concerns with pre-signed URLs, we can ensure:
Upvotes: 0
Reputation: 714
If you want them to load data into a bucket, the best way will almost always be the pre-signed URL. This gives you complete control over how you hand out access to the bucket, but also allows them to directly upload into the bucket when they have the access.
In the first two examples the user can send malicious data to your API, potentially DOSing the server / incurring costs on you to manage the payloads as you have no control over access (it is public).
In the third case they can request a URL from you, but that is it, other than spamming you for requests for URLs, unless you grant them a URL they can't access the bucket or do anything else. This seems much better than spamming your upload with large junk files and having you process them before you decide you didn't want them anyway.
Finally using the pre-signed URL is the pattern AWS would expect you to use, and so have a lot of support for managing the access, roles, logging and monitoring etc that you would want to put around this service. When you are standing up the API yourself this will all be up to you to manage.
Upvotes: 2