Resuming a Failed HTTP Download With Qt & QNetworkRequest

Question

I'm trying add auto-update capabilities to an application I'm developing. I've based this functionality off the Qt HTTP Example (and by based I mean I copied this example exactly and then went from there). It's downloading a ZIP file and then extracting its contents to patch the application.

Occasionally, when downloading, the connection will fail, and the download stops. To be a little more user friendly, I figured I'd add auto-restart capabilities to the downloader, where it will attempt to restart the download once if it fails.

Here are the highlights of my code - the method names match the method names in the example:

void Autopatcher::httpReadyRead()
{
    //file is a QFile that is opened when the download starts
    if (file) {
        QByteArray qba = reply->readAll();
        //keep track of how many bytes have been written to the file
        bytesWritten += qba.size();
        file->write(qba);
    }
}

void Autopatcher::startRequest(QUrl url)
{
    //doResume is set in httpFinished() if an error occurred
    if (doResume) {
        QNetworkRequest req(url);
        //bytesWritten is incremented in httpReadyRead()
        QByteArray rangeHeaderValue = "bytes=" + QByteArray::number(bytesWritten) + "-";
        req.setRawHeader("Range",rangeHeaderValue);
        reply = qnam.get(req);
    } else {
        reply = qnam.get(QNetworkRequest(url));
    }
    //slot connections omitted for brevity
}

//connect(reply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(fileGetError(QNetworkReply::NetworkError)));
void Autopatcher::fileGetError(QNetworkReply::NetworkError error) {
    httpRequestAborted = true;
}

void Autopatcher::httpFinished() {
    //If an error occurred
    if (reply->error()) {
        //If we haven't retried yet
        if (!retried) {
            //Try to resume the download
            doResume=true;
            //downloadFile() is a method that handles some administrative tasks
            //like opening the file if doResume=false
            //and calling startRequest() with the appropriate URL
            QTimer::singleShot(5000,this,SLOT(downloadFile()));
        }
        //If we have retried already
        else {
            //Give up :(
            if (file) {
                file->close();
                file->remove();
                delete file;
                file = 0;
            }
        }
    //If no error, then we were successful!
    } else {
        if (file) {
            file->close();
            delete file;
            file = 0;
        }
        //Apply the patch
        doPatch();
    }
    reply->deleteLater();
    reply = 0;
}

Now, if the download completes normally with no interruptions, it works just fine. The ZIP extracts perfectly. However, if the connection fails and the application restarts the download, it does finish downloading, and I can see all the contents of the ZIP file in 7-zip, but I cannot extract them (7-zip said something along the lines of "tried to move the pointer before the start of the file).

I'm assuming that I've made a simple off-by-one error somewhere, like in the HTTP Range header. I've seen an example of how to pause & resume downloads at this blog, but he writes the contents of the stream to file at pause, whereas I stream them into the file in httpReadyRead. I don't know if that's causing a problem.

For testing, I've been using Sysinternals TCPView to sever the TCP connection during download. I'm not sure how to debug this further, so let me know if more information would be useful!

Answer 1

So today I investigated deeper. I was originally thinking that the file sizes of the uninterrupted and interrupted versions were the same +- a few bytes, but I was wrong. I downloaded two versions of the file, and the sizes were off by around 2 megabytes.

So, I compared them using VBinDiff (a nice utility if you're not afraid of a console interface) and here's what I found:

• The files stopped matching at address 0x0154 21F3.
• That address in the bad file matched address 0x0178 1FD3 in the good file, and they continued to match until the end.
• Therefore, the bad file was missing 2,358,752 bytes - which matches the 2MB approximation I was seeing in Explorer.

This confirmed that when I tried to restart a download, I was skipping a significant portion of the remote file. Unsure of what was going on, I decided to check the value of bytesWritten, which I was using to track how many bytes I had written into the file. This value was what I was writing into the Range Request Header, so its value had to be incorrect. (See the httpReadyRead() function in the question).

So I added the code below, right before setting the Range Request Header:

file->flush();
bytesWritten = file->size();

Debugging the code, I was surprised to find that

bytesWritten = 28,947,923
file->size() = 26,589,171

Confirming that the bytesWritten value was incorrect. In fact, when I used the file size, instead of the bytesWritten value, the download was able to restart and finish successfully!

I'm not going to go any deeper, as this works for me. In fact, this would allow restarting downloads between instances of the application, so in my opinion this is a superior method.

tl;dr Don't keep track of the bytes written to the file. Just check the file size when restarting a failed download.