How do I zip on the fly and stream to Response.Output in real time?

Question

I am trying to use the following code: I get a corrupted zip file. Why? The file names seem OK. Perhaps they are not relative names, and that's the problem?

      private void trySharpZipLib(ArrayList filesToInclude)
    {
        // Response header
        Response.Clear();
        Response.ClearHeaders();
        Response.Cache.SetCacheability(HttpCacheability.NoCache);
        Response.StatusCode = 200; // http://community.icsharpcode.net/forums/p/6946/20138.aspx
        long zipSize = calculateZipSize(filesToInclude);
        string contentValue = 
            string.Format("attachment; filename=moshe.zip;"
                          ); // + " size={0}", zipSize);
        Response.ContentType = "application/octet-stream"; //"application/zip"; 
        Response.AddHeader("Content-Disposition", contentValue);
        Response.Flush();

        using (ZipOutputStream zipOutputStream = new ZipOutputStream(Response.OutputStream) ) 
        {
            zipOutputStream.SetLevel(0);

            foreach (string f in filesToInclude)
            {
                string filename = Path.Combine(Server.MapPath("."), f);
                using (FileStream fs = File.OpenRead(filename))
                {
                    ZipEntry entry =
                        new ZipEntry(ZipEntry.CleanName(filename))
                            {
                                DateTime = File.GetCreationTime(filename),
                                CompressionMethod = CompressionMethod.Stored,
                                Size = fs.Length
                            };
                    zipOutputStream.PutNextEntry(entry);

                    byte[] buffer = new byte[fs.Length];
                    // write to zipoutStream via buffer. 
                    // The zipoutStream is directly connected to Response.Output (in the constructor)
                    ICSharpCode.SharpZipLib.Core.StreamUtils.Copy(fs, zipOutputStream, buffer); 
                    Response.Flush(); // for immediate response to user
                } // .. using file stream
            }// .. each file
        }
        Response.Flush();
        Response.End();
    }

Answer 1

Boy, that's a lot of code! Your job would be simpler using DotNetZip. Assuming a HTTP 1.1 client, this works:

Response.Clear();
Response.BufferOutput = false;
string archiveName= String.Format("archive-{0}.zip", DateTime.Now.ToString("yyyy-MMM-dd-HHmmss"));
Response.ContentType = "application/zip";
// see http://support.microsoft.com/kb/260519
Response.AddHeader("content-disposition", "attachment; filename=" + archiveName);  
using (ZipFile zip = new ZipFile())
{
    // filesToInclude is a IEnumerable<String> (String[] or List<String> etc)
    zip.AddFiles(filesToInclude, "files");
    zip.Save(Response.OutputStream);
}
// Response.End(); // will throw an exception internally.
// Response.Close(); // Results in 'Failed - Network error' in Chrome.
Response.Flush(); // See https://stackoverflow.com/a/736462/481207
// ...more code here...

If you want to password-encrypt the zip, then before the AddFiles(), insert these lines:

    zip.Password = tbPassword.Text; // optional
    zip.Encryption = EncryptionAlgorithm.WinZipAes256; // optional

If you want a self-extracting archive, then replace zip.Save() with zip.SaveSelfExtractor().

---

Addendum; some people have commented to me that DotNetZip is "no good" because it creates the entire ZIP in memory before streaming it out. This isn't the case. When you call AddFiles, the library creates a list of entries - objects that represent the state of the things to be zipped up. There is no compression or encryption done until the call to Save. If you specify a stream to the Save() call, then all the compressed bytes get streamed directly to the client.

In the SharpZipLib model, it's possible to create an entry, then stream it out, then create another entry, and stream it out, and so on. With DotNetZip your app creates the complete list of entries first, and then streams them all out. Neither approach is necessarily "faster" than the other, though for long lists of files, say 30,000, the time-to-first-byte will be faster with SharpZipLib. On the other hand I would not recommend dynamically creating zip files with 30,000 entries.

---

EDIT

As of DotNetZip v1.9, DotNetZip supports a ZipOutputStream as well. I still think it's simpler to do things the way I've shown here, though.

---

Some people have the case where they have zip content that is "mostly the same" for all users, but there's a couple of files that are different for each one. DotNetZip is good at this, too. You can read in a zip archive from a filesystem file, update a few entries (add a few, remove a few, etc), then save to Response.OutputStream. In this case DotNetZip does not re-compress or re-encrypt any of the entries that you haven't changed. Much faster.

Of course DotNetZip is for any .NET app, not only ASP.NET. So you can save to any stream.

If you want more info, check out the site or post on the dotnetzip forums.

Answer 2

Necromancing.
Here's how it's done properly, using closures, starting with DotNetZip v1.9+ as recommended by Cheeso in the comments:

public static void Run()
{
    using (Ionic.Zip.ZipFile zip = new Ionic.Zip.ZipFile())
    {

        for (int i = 1; i < 11; ++i)
        {
            zip.AddEntry("LeaseContractForm_" + i.ToString() + ".xlsx", delegate(string filename, System.IO.Stream output)
            {
                // ByteArray from ExecuteReport - only ONE ByteArray at a time, because i might be > 100, and ba.size might be > 20 MB
                byte[] ba = Portal_Reports.LeaseContractFormPostProcessing.ProcessWorkbook();
                output.Write(ba, 0, ba.Length);
            });
        } // Next i 

        using (System.IO.Stream someStream = new System.IO.FileStream(@"D:\test.zip", System.IO.FileMode.Create, System.IO.FileAccess.Write, System.IO.FileShare.None))
        {
            zip.Save(someStream);
        }
    } // End Using zip 

} // End Sub Run 

And the VB.NET variant, in case anybody needs it (please note that this is just a test; in reality it would be called with different in_contract_uid and in_premise_uid for each step in the loop):

Imports System.Web
Imports System.Web.Services


Public Class test
    Implements System.Web.IHttpHandler


    Sub ProcessRequest(ByVal context As HttpContext) Implements IHttpHandler.ProcessRequest
        Dim in_contract_uid As String = context.Request.Params("in_contract_uid")
        Dim in_premise_uid As String = context.Request.Params("in_premise_uid")

        If String.IsNullOrWhiteSpace(in_contract_uid) Then
            in_contract_uid = "D57A62D7-0FEB-4FAF-BB09-84106E3E15E9"
        End If

        If String.IsNullOrWhiteSpace(in_premise_uid) Then
            in_premise_uid = "165ECACA-04E6-4DF4-B7A9-5906F16653E0"
        End If

        Dim in_multiple As String = context.Request.Params("in_multiple")
        Dim bMultiple As Boolean = False

        Boolean.TryParse(in_multiple, bMultiple)


        If bMultiple Then
            Using zipFile As New Ionic.Zip.ZipFile

                For i As Integer = 1 To 10 Step 1
                    ' Dim ba As Byte() = Portal_Reports.LeaseContractFormReport.GetLeaseContract(in_contract_uid, in_premise_uid) '
                    ' zipFile.AddEntry("LeaseContractForm_" + i.ToString() + ".xlsx", ba) '

                    zipFile.AddEntry("LeaseContractForm_" + i.ToString() + ".xlsx", Sub(filename As String, output As System.IO.Stream)
                                                                                        Dim ba As Byte() = Portal_Reports.LeaseContractFormReport _
                                                                                        .GetLeaseContract(in_contract_uid, in_premise_uid)
                                                                                        output.Write(ba, 0, ba.Length)
                                                                                    End Sub)
                Next i

                context.Response.ClearContent()
                context.Response.ClearHeaders()
                context.Response.ContentType = "application/zip"
                context.Response.AppendHeader("content-disposition", "attachment; filename=LeaseContractForm.zip")
                zipFile.Save(context.Response.OutputStream)
                context.Response.Flush()
            End Using ' zipFile '
        Else
            Dim ba As Byte() = Portal_Reports.LeaseContractFormReport.GetLeaseContract(in_contract_uid, in_premise_uid)
            Portal.ASP.NET.DownloadFile("LeaseContractForm.xlsx", "attachment", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", ba)
        End If

    End Sub


    ReadOnly Property IsReusable() As Boolean Implements IHttpHandler.IsReusable
        Get
            Return False
        End Get
    End Property


End Class

Answer 3

For those who would miss the ZipOutputStream of SharpZipLib, here is a simple code that makes it possible to use DotNetZip the "regular .NET stream way".

However, take note that it is inefficient compared to a real on-the-fly streaming solution like SharpZipLib does, as it uses an internal MemoryStream before actually calling the DotNetZip.Save() function. But unfortunately, SharpZibLib does not sports EAS Encryption yet (and certainly never). Let's hope that Cheeso will add this functionality in dotNetZip any time soon ? ;-)

/// <summary>
/// DotNetZip does not support streaming out-of-the-box up to version v1.8.
/// This wrapper class helps doing so but unfortunately it has to use
/// a temporary memory buffer internally which is quite inefficient
/// (for instance, compared with the ZipOutputStream of SharpZibLib which has other drawbacks besides).
/// </summary>
public class DotNetZipOutputStream : Stream
{
    public ZipFile ZipFile { get; private set; }

    private MemoryStream memStream = new MemoryStream();
    private String nextEntry = null;
    private Stream outputStream = null;
    private bool closed = false;

    public DotNetZipOutputStream(Stream baseOutputStream)
    {
        ZipFile = new ZipFile();
        outputStream = baseOutputStream;
    }

    public void PutNextEntry(String fileName)
    {
        memStream = new MemoryStream();
        nextEntry = fileName;
    }

    public override bool CanRead { get { return false; } }
    public override bool CanSeek { get { return false; } }
    public override bool CanWrite { get { return true; } }
    public override long Length { get { return memStream.Length; } }
    public override long Position
    {
        get { return memStream.Position; }
        set { memStream.Position = value; }
    }

    public override void Close()
    {
        if (closed) return;

        memStream.Position = 0;
        ZipFile.AddEntry(nextEntry, Path.GetDirectoryName(nextEntry), memStream);
        ZipFile.Save(outputStream);
        memStream.Close();
        closed = true;
    }

    public override void Flush()
    {
        memStream.Flush();
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        throw new NotSupportedException("Read");
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        throw new NotSupportedException("Seek");
    }

    public override void SetLength(long value)
    {
        memStream.SetLength(value);
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        memStream.Write(buffer, offset, count);
    }
}

Answer 4

Not quite sure how to do this in ASP.NET (haven't tried it before), but in general if the HTTP client supports HTTP v1.1 (as indicated by the version of its request), the server can send a 'Transfer-Encoding' response header that specifies 'chunked', and then send the response data using multiple data blocks as they become available. This allows for real-time streaming of data where you don't know the final data size ahead of time (and thus cannot set the 'Content-Length' response header). Have a look at RFC 2616 Section 3.6 for more details.

Answer 5

Have you tried flushing the ZipOutputStream before flushing the response? Can you save the zip on the client and test it in a zip utility?

Answer 6

Try adding the following header.

Response.AddHeader("Content-Length", zipSize);

I know that was causing me issues before.

Edit:

These other 2 may help as well:

Response.AddHeader("Content-Description", "File Transfer");
Response.AddHeader("Content-Transfer-Encoding", "binary");