Requesting advice for accepting large file uploads

Question

I'm looking at building a web app that includes a file upload element. I'd like users to be able to upload files of any type and of fairly large size (say, up to 100MB). This will be a publicly accessible site, so security is obviously very important.

I've done a decent amount of googling in search of answers, but it's difficult when I don't really know exactly what I'm searching for.

My experience is mainly with PHP, but I realise that PHP is not considered to be the best when it comes to file uploading, so I'm happy to look at other languages if necessary. Although, if a decent solution using PHP can be acheived, that would be preferable.

As I have no experience with this kind of project, I'm also fairly in the dark on what kind of server setup is required for such an app.

I have braistormed a few ideas, but am willing to budge on them if unreasonable:

• I'd like to use Amazon S3 to store the files if possible (to reduce the load on the server)
• I'd like to be able to rename the files after upload
• I'm considering Uploadify (uploadify.com) for the client side

Basically, imagine I was looking to build a file-sending app like wetransfer.com or yousendit.com and you'll get the general idea.

I'm familiar with all the usual PHP file upload issues (checking mime-types, upload_max_filesize, memory_limit, etc, etc) covered by 99% of posts on the internet on this topic, but obviously this project goes a fair bit beyond your average, run-of-the-mill avatar upload script.

I know this is a massive topic and I'm obviously not expecting anyone to present me with a magic solution, but basically I'm looking for some pointers on where to start. Can anyone recommend any good books, articles or websites where I can gain a better understanding of the requirements of the task? Covering everything from the programming to the server requirements? Even if it's just a list of keywords or phrases that I should be googling.

Thanks in advance!

P.S. I wasn't 100% sure if this was the right StackExchange site to post this question on. I also considered serverfault.com and webmasters.stackexchange.com. If you think this question would be better asked elsewhere, please let me know.

Answer 1

If you funnel the upload through your PHP you need to make sure that it accepts those large files. Especially upload_max_filesize, post_max_size and max_input_time. See POST method uploads for a general how to.

With Resumable.js you could circumvent above limitations quite nicely. It uploads small chunks of your 100MB at a time. This allows it to keep track of what's been uploaded to allow pause/resuming uploads.

While I've never worked with Amazon S3, I do not believe you can upload data from any client - at least not without some sort of authentication. You'll probably have to funnel the upload through your own server in order to push it to S3.