Qt C++ Application: Dealing with std::bad_alloc Error When Downloading Large Files

Question

I'm developing a Qt C++ application that involves downloading and processing large files from a remote server. However, I've encountered a persistent issue with memory allocation errors, specifically std::bad_alloc, during the download process.

Here are some key details about my application and the problem:

I'm using Qt's QNetworkAccessManager for file downloads. The issue seems to occur when downloading a specific file that is approximately 363MB in size. Other files of smaller sizes download without any problems. The error is consistently thrown when attempting to allocate memory for this particular file.

Whenever I attempt to download this specific large file, my application crashes with a std::bad_alloc error, indicating a memory allocation problem. I've tried various approaches to handle memory efficiently, including using QByteArray and QIODevice for chunked reading and writing, but the issue persists.

Chunked reading and writing using QByteArray. Using QIODevice without invoking readAll() to avoid loading the entire file into memory at once. Implementing deleteLater() for resource cleanup. Checking system memory availability, which appears to be sufficient.

This is request part

void NetworkManager::downloadDataFromServer(JsonManager *jdoc){
//    QQueue<QNetworkRequest> downloadQueue;
    QString base = jdoc->baseUrl;
//    QNetworkRequest request;
    m_totalFileCount = jdoc->files.size();

    for (int i = 0; i < m_totalFileCount; i++) {
        QJsonObject jsonObject = jdoc->files.at(i).toObject();
        QUrl reqUrl(base + jsonObject.value("url").toString());
        QNetworkRequest areq;
        areq.setUrl(reqUrl);

        QNetworkReply *reply = assetManager->get(areq);
        qDebug()<<"requesting:"<<jsonObject.value("url").toString();
        // Connect each reply's finished signal to a slot for handling it
        connect(reply, &QNetworkReply::finished, this, &NetworkManager::onAssets);
    }

}

And this is slot

void NetworkManager::onAssets()
{
    QNetworkReply *reply = qobject_cast<QNetworkReply*>(sender());
    if(reply->url().toString().contains("91c11a970454232586c0bc9903af9505"))
    {
        qDebug()<<"hereerereeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee";
    }

    if (!reply) {
        // Handle the case where sender() did not return a valid reply
        return;
    }

    // If an error occurs in the process of obtaining data
    if(reply->error()){
        // We inform about it and show the error information
        qDebug() << "ERROR";
        qDebug() << reply->errorString();
        emit error(reply->errorString());
    } else {
        QString filePath = QCoreApplication::applicationDirPath() + "/app/"+reply->url().toString().replace(baseUrl,"");
        QFileInfo fileInfo(filePath);
        QDir dir(fileInfo.absolutePath());

        if (!dir.exists()) {
            if (dir.mkpath(fileInfo.absolutePath())) {
//                qDebug() << "Directory structure created: " << fileInfo.absolutePath();
            } else {
                qDebug() << "Failed to create directory structure: " << fileInfo.absolutePath();
            }
        }
        QFile file(filePath);
        if (file.open(QIODevice::WriteOnly))
        {
            QIODevice* source = reply; // No readAll() here
            while (!source->atEnd()) {
                QByteArray buffer = source->read(8192); // Read a chunk
                file.write(buffer); // Write the chunk to the file
            }
            file.close();
            m_downloadedFileCount++;
            float percent =(float) m_downloadedFileCount/(float)m_totalFileCount;
            emit downloadedFileCountChanged(percent, filePath);
        }
        else
        {
            qDebug() << "Failed to create or save asset file.";
        }
    }
    reply->deleteLater();
}