Best way to receive file through webservice

Question

I have an API consisting of ASP.NET webservices callable through GET/POST/SOAP.

A new functionality will require me to accept files through this webservice. Basically I'm allowing users to upload files to a file archive through the API, which they'd otherwise do by logging into our browser based administration system.

The API needs to be as easily consumed as possible. I'm thinking of dropping SOAP support since our PHP clients mainly use the GET/POST methods, and the .NET clients don't mind either way. With that as a prerequisite, I'm thinking of simply creating an UploadFile(fileName) method, and the requiring the user to send the file through POST as a normal file upload.

However, I will not be able to specify the file field as a parameter to the method, right? So if I simply state in the documentation "file should be sent in POST field called 'File'", would that pose any problems when I need to read the file?

All files are in binary format, PDF's, various image formats, FLV and so forth. Also, file sizes will mainly be in the 2-20MB vicinity, but given the above solution, would I have any troubles accepting files in the 250MB area?

Receiving a file this way would result in the file being loaded completely into memory, before I write it to disk - is there any way around this, besides letting the service receive a Stream and thus disabling me from accepting other parameters, and hindering the easy usage of the service?

---

Besides what's possible on my side, I'm also curious in regards to how I make it as easy as possible for the callees to send the file. I'm guessing POST is one of the most accessible ways of receiving the file, but if anyone has any comments, I'd like to hear them as well.

Answer 1

However, I will not be able to specify the file field as a parameter to the method, right? So if I simply state in the documentation "file should be sent in POST field called 'File'", would that pose any problems when I need to read the file?

There's no reason you can't include additional POST (or GET, for that matter) parameters. I would probably take the filename as an additional POST parameter myself, so that clients that automatically use the local filename in the request can rename easily. If I expected naming conflicts, I may even return a server chosen name in the event of conflicts...I think the REST folks use some sort of HTTP Created status code for that.

All files are in binary format, PDF's, various image formats, FLV and so forth. Also, file sizes will mainly be in the 2-20MB vicinity, but given the above solution, would I have any troubles accepting files in the 250MB area?

You'll need to tweak the maxRequestLengths and executionTimeouts in web.config. And, if anyone tries a 250MB upload over dialup - well, let's just say it probably won't work.

Receiving a file this way would result in the file being loaded completely into memory, before I write it to disk - is there any way around this, besides letting the service receive a Stream and thus disabling me from accepting other parameters, and hindering the easy usage of the service?

ASP.NET 2.0+ spools uploaded files to disk as they come in, to avoid preloading the entire request into RAM. Tweak web.config's requestLengthDiskThreshold as appropriate to balance speed and memory usage. You couldn't take a Stream if you wanted to...though you may be able to take in a byte[] array. I'm really not sure what that'd mean though....

For small files, you could do base 64 which should be trivial for clients. But, for 20MB - the overhead wouldn't be worth it.

There's a bunch of different ASP.NET file upload components - which basically hook up to Begin_Request as an HttpModule, and intercept the request stream for any multippart/form-uploads. With ASP.NET 2.0, this is probably unnecessary, unless you want to provide some sort of progress callback or something. If that's the case - the mechanics of them should work with a web service the same.

RFC 1867 is the relevant spec, but this upload component does a good job of laying out what the request looks like if you're intent on parsing your own.

The one question mark I would have, is what the level of support for multipart/form-data in the http libraries of your clients is. I'd suspect that it's pretty good all around, but you may want to check.

Answer 2

Not a direct answer but you may also want to look into BITS - Background Intelligent Transfer Service [wiki]

It may or may not be relevant to you / your customers.

Answer 3

"would that pose any problems when I need to read the file?" No problems. When you handle the POST request, you have a MIME attachment to the POST, which is the file. It's relatively straightforward to read this file.

"would I have any troubles accepting files in the 250MB area?" No. Except that it takes a some time to transfer big files.

"Receiving a file this way would result in the file being loaded completely into memory" Not necessarily true. Many web servers process the request up to the MIME attachment and stop reading, making the HTTP headers, the request and the open socket available for further processing. I don't know about ASP, but I do know that almost all frameworks avoid reading the data. The framework leaves that to your application.

You can, consequently, read the data in chunks and write (or process) those chunks without reading the entire thing into memory.

POST is precisely how this is done. There aren't a lot of alternatives without stepping outside the narrow HTTP web services space. You can, of course, use other protocols, like FTP (TFTP, SFTP) to move files around, but that's often confusing for folks writing client apps.

You can, if you want, provide a client library that creates a HttpWebRequest object with the POST method, and a stream with the attachment. But you wouldn't be saving much, as the HttpWebRequest class looks pretty simple. Here's a code sample.

Answer 4

There's nothing horribly wrong with having a simple POST method for submitting a file, just the detail of tying the file upload to any pertinent meta data (i.e. the other parameters to the method if it were bundled with the method).

Today, I believe the way for handle large objects in SOAP calls, however, is MTOM. This is an optimization, but it gives you a stream interface to the actual payloads. So, the framework can cache the large object to disk if necessary readily (whether they actually do is another question, but no one wants to have to absorb a 250M upload in to memory, so I imagine any modern one will offload the large attachments).