CODEWITHSUNDEEP
c#jquerywcf
Jake Woods
Jake Woods

Reputation: 1828

How can I create a WCF service that can accept large files of differing formats from a web based client

We have a web service that needs to provide the client the ability to upload large files that are then parsed by one of the available mechanisms which is selected by the client. Typical usage of this service will involve uploading a large file and specifying the parsing mechanism to use, the server will then parse the uploaded data and store it in a database.

For example, we have a number of excel files containing useful information. Unfortunately there are multiple different formats and we need to be able to parse them all. Our currently solution asks that when the user uploads a excel file they also select from a list of predefined parsing methods.

Currently the service provides one method with the following signature:

[OperationContract, WebInvoke(UriTemplate = "/DoUpload/{fileType}")]
void DoUpload(string fileType, Stream fileData);

We need some way of allowing the web based client (which leverages jQuery) to invoke DoUpload such that fileType is a client selected string and fileData is a file uploaded from the client.

Ideally this mechanism would also allow us to expose a list of valid fileTypes from the same service.

Other information:

Upvotes: 0

Views: 2522

Answers (1)

Jake Woods
Jake Woods

Reputation: 1828

WCF supports two methods of uploading, streaming and buffered. Buffered is the default mode and involves buffering the entire file and sending it in one big chunk to the server. This works well for small to medium files but tends to be far too slow for large files. Streaming sends bits of the file over a number of responses to the server and has a number of benefits such as the ability to resume the stream if it is interrupted.

Obviously the appropriate method for this problem is streaming, however, because streaming isn’t the default we need to do some configuration work in order to get WCF to use it. This answer applies to a WCF Service in particular as enabling streaming involves modifying Web.config

So to start with, let’s assume you have a single exposed method that accepts a stream, something like this:

[OperationContract]
[WebInvoke(UriTemplate = "/Upload", Method = "POST")]
void Upload(Stream data);

We need to tell WCF to use streaming for this endpoint, to do this we need a binding that allows streaming, we can use something like this in Web.config:

<configuration>
    ...
    <bindings>
        <webHttpBinding>
            <binding name="httpStreamingBinding" transferMode="Streamed" />
        </webHttpBinding>
    </bindings>
    ...
    <services>
        <service name="MyServiceNamespace.MyServiceName">
            <endpoint address="" behaviorConfiguration="web" binding="webHttpBinding"
                      bindingConfiguration="httpStreamingBinding" name="UploadEndpoint"
                      contract="MyServiceNamespace.IMyServiceName" />
        </service>
    </services>
    ...
    <behaviors>
        ...
        <endpointBehaviors>
            <behavior name="web">
                <webHttp />
            </behavior>
        </endpointBehaviors>
    </behaviors>
</configuration>

This creates an endpoint that points to the class containing the Upload method and configures it to use streaming transfers. But we've still got a problem, large files take time to transfer and the web server default timeout is too low to transfer big files, we need to modify more of our Web.config.

First we need to change the timeout length and max received message size for the httpStreamingBinding like so (maxRecievedMessageSize is in bytes):

<binding name="httpStreamingBinding" maxReceivedMessageSize="4294967296"
         transferMode="Streamed" 
         crossDomainScriptAccessEnabled="true"
         openTimeout="00:01:00"
         closeTimeout="00:01:00"
         receiveTimeout="02:00:00"
         sendTimeout="02:00:00"
         />

Then we need to modify our http runtime to accept large files, here I've chosen 4gb as the maximum (maxRequestLength is in kb):

<system.web>
    ...    
    <httpRuntime
      executionTimeout="7200" 
      maxRequestLength="4194304" />
</system.web>

Now we can recieve streamed data, however there's still more to do. Typically a large HTTP upload is done using multipart/form-data content type, this means the data you recieve on the stream will not just be the uploaded file, but also additional data from the form. You can parse this data yourself manually or you can use an existing parser. Lorenzo provides an excellent multipart data parser in this answer

We've got another problem though, uploading multipart data is simple using a HTML form, but what if we want to use javascript? Depending on the browser there's various levels of support for uploading streamed data, however the jQuery file uploader plugin provides excellent support for uploading files to a streamed data service. Just make sure to use "files[]" as the file name for the multipart parser like so:

[OperationContract]
[WebInvoke(UriTemplate = "/Upload", Method = "POST")]
void Upload(Stream data)
{
    var parser = new HttpMultipartParser(data, "files[]");
    ...
}

Now that we've got the data, we need some way to do some logic based on the type of file uploaded. I solved this by changing the parsing method based on a URL parameter, for example /Upload/FormatOne would use the FormatOne method while /Upload/FormatTwo would use FormatTwo. This is accomplished using this method:

delegate void FileFormatHandler(Stream data);

[OperationContract]
[WebInvoke(UriTemplate = "/Upload/{fileType}", Method = "POST")]
void Upload(string fileType, Stream data)
{
    var parser = new HttpMultipartParser(data, "files[]");

    FileFormatHandler handler = selectHandler(fileType);
    handler(data);
}

Unfortunately, this method means that the service will not longer work with the SOAP based WCF calling mechanism as Stream is supposed to be the only argument in SOAP, however a web call will be unaffected by this restriction.

We now have a WCF service that accepts large files using streaming and is able to invoke different parsing methods based on what URL the user uses. To upload data the user simply POST's some information to the appropriate URL. For example, if the user wanted to upload a file and have it parsed by the MyFancyFormat parser they would POST to the following URL:

http://myserver/MyService.svc/Upload/MyFancyFormat

Upvotes: 2

Related Questions