CODEWITHSUNDEEP
c++qt
Evan Levesque
Evan Levesque

Reputation: 3213

Qt Multi-Threading

I am Trying to read data from text files and save tokens in SQLite databae running on windows The Files size is about 300MB , the thing is that I am not using threads yet , the application collapses and stuck with "not responding" message here is my code

QDir dir(ui->lineEdit->text());
if(dir.exists() && ui->lineEdit->text()!=""){
    CreateTables();

    dir.setFilter(QDir::Files | QDir::Hidden | QDir::NoSymLinks);
    dir.setSorting(QDir::Size | QDir::Reversed);

    QFileInfoList list = dir.entryInfoList();
    for (int i = 0; i < list.size(); ++i)  {
        QFileInfo fileInfo = list.at(i);
        QFile file(ui->lineEdit->text()+"/"+ fileInfo.fileName());

        if (file.open(QIODevice::ReadOnly | QIODevice::Text)) {
            QSqlQuery qrry;
            qrry.prepare( "INSERT INTO documents (path) VALUES ('"+ui->lineEdit->text()+"/"+ fileInfo.fileName()+"')" );
            qrry.exec();

            QString line;
            QTextStream in(&file);
            int lineCount=0;
            while (!in.atEnd()) {

                line = in.readLine();
                lineCount++;
                QRegExp rx("(\\ |\\,|\\.|\\:|\\t)"); 
                QStringList line_tokens = line.split(rx);

                for(int i=0;i<line_tokens.length();i++){
                    if(line_tokens[i].length()>3){
                        QSqlQuery qry;
                        qry.prepare( "INSERT INTO tokens (token,path,line) VALUES ('"+line_tokens[i]+"', '"+ui->lineEdit->text()+"/"+ fileInfo.fileName()+"','"+QString::number(lineCount)+"')" );
                        qry.exec();
                    }
                }
            }
        }
        else{
            QMessageBox::information(this,"File Read Error","Couldn't Open File, Please Make Sure That This File Is Accessable And Readable !");
        }
        std::cout << std::endl;
    }
}
else{
    QMessageBox::critical(this,"Directory choosen is Not Valid","Please Make Sure That You Have Choosen A Valid Directory!");
}

it works for a small file sizes but when it comes to huge amount of data it becomes gives the "not responding" message

will threads save the application from being collapsed ?? and how may I implements threads to do this job for more thant 20 files of 15 MB each

Upvotes: 1

Views: 928

Answers (2)

zar
zar

Reputation: 12247

This is ideal situation where you would want to use threads. The threads will do all the work in the background and while it will still take its time to process the data, your GUI will not appear hanged or not responding.

I don't fully understand your application but you could either write a single thread which does all this processor intensive work or even two threads one of reading file and the other for database but two threads require greater effort to implement. I will suggest stick with one thread first as it's simpler.

You definitely want to use threads here, it will solve your application not responding message which often people interpret as 'it collapsed' or 'hanged' while the application main thread is just too busy depriving GUI to function properly.

Upvotes: 0

Lol4t0
Lol4t0

Reputation: 12557

First, you can simply... wait until it process all the data. If you should run this program only once, it is ok.

Then, you should not generate such stream of requests to database. It is much better to run one single big insertion, than a lot of small. You can use batch execution.

Then, it's probably no reason of trying to parallel writing to Db. I think, it's smart enough to manage writing efficiently.

But it worth paralleling parsing. The best solution here is to use QtConcurrent framework. It is made exactly for such tasks.

Upvotes: 1

Related Questions