Calling service to process large files

Question

So I have this WCF service that provides me with PDF processing functions. Right now it is accepting and returning plain byte arrays (byte[]) but this means it is always reading everything to memory and even when it process streams it has to ToArray() them to byte array. Problem is, that is causing out of memory problems and heap fragmentation.

I have two choices to optimize it so it won't use so much memory:

• Use streams. Although tempting, I may have to persist it to disk to be processed by the PDF toolkit. It is also a very finicky strategy given the limited number of bindings and also restrictions on duplex streaming and number of parameters.
• Pass UNC file paths back and forth. Seems promising but complicates things because someone needs to cleanup the files after use.

Which alternative would achieve best results in terms of optimizing resource usage (memory, network, file system)?

Answer 1

I would go for the UNC file path option as well. I have used this when transferring user-uploaded files from a Web Server into SharePoint via an Application Server. The Web Server would pass the file path to the App Server which would then pick up the file and load it into SP. It was a much better option that passing the file as a byte array via a web service call from the Web Server to the App Server.

Regards "someone needs to cleanup files after", if you are concerned that your service may not alway do the cleanup, e.g. if there was an exception, you can always have a backup plan and use something simple like a powershell script with a scheduled task to delete old files that should have been deleted earlier.

One thing to consider though is the chance of filename clashes. In our case it was feasible for two users to upload separate files with the same name so we had to ensure that the right file was put into the right SP folder. We solved this by having the web server that stored the file locally, rename the file with a GUID name and then passing the new name to the App Server. The original filename was sent as metadata to the App Server so that it could rename the file back to its original name before loading into SP.

Answer 2

It is old school, but I would go with "Pass UNC file paths back and forth". Fortunately, we developers have had a few years of experience with this approach (see: print spooler in Unix Version 7 around 1980 or so).