CODEWITHSUNDEEP
c++multithreading
WhoAmI
WhoAmI

Reputation: 643

multiple threads for vector processing

I have 2 vectors with some filenames and function that process this files:

vector<string> vecFilenames1; // {filename1_1, filename1_2, filename1_3, ...}
vector<string> vecFilenames2; // {filename2_1, filename2_2, filename2_3, ...}

This vectors have same size. How I process now:

// function for processing
void doSomeStuff() {// ...}
// processing loop
for (int i = 0; i < vecFilenames1.size();i++) {
    doSomeStuff(vecFilenames1[i], vecFilenames2[i]);
}

I have 4 threads (2 cores) and I want do such a process faster, how could I do that?

EDIT 1

I use mingw compiler:

g++ (MinGW.org GCC-8.2.0-5) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.

Do I need to change it to newer version for easy solving my problem?

EDIT 2

I updated my gcc:

g++.exe (MinGW.org GCC Build-2) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.

Upvotes: 1

Views: 482

Answers (1)

Alex Guteniev
Alex Guteniev

Reputation: 13739

You should partition vector into ranges and process each range in a threadpool thread.

C++17 parallel algorithms is an easy way to accomplish this. By using std algorithm, you don't need to do things like partitioning vector, and calling thread pool manually.

You can use Intel TBB library or Open MP directives to achieve similar without C++17 support.

Or roll your own implementation. std::async is to run a thread pool task, std::hardware_concurrency to get the estimate on number of cores

Example of parallel for_each:

#include <algorithm>
#include <chrono>
#include <iostream>
#include <execution>
#include <mutex>
#include <string>
#include <thread>

using namespace std;

vector<string> vecFilenames1;
vector<string> vecFilenames2;

int main() {
    for (int i = 1; i < 1000; i++)
    {
        vecFilenames1.push_back("filename1_" + to_string(i));
        vecFilenames2.push_back("filename2_" + to_string(i));
    }

    mutex m;

    auto f = [&](const string& fn1)
    {
        // Comupute other element via pointer arthimetics
        // Works only with vector, for, say, deque, use container of struct
        const string& fn2 = vecFilenames2[&fn1 - vecFilenames1.data()];

        // simulate processing (to hide mutex unfairness and make threads
        // working concurrently)
        // replace with read processing
        using namespace chrono_literals;
        this_thread::sleep_for(30ms);

        // avoid doing any real work under the lock to benefit from paralleling
        lock_guard<mutex> guard(m);

        // ideally don't do console i/o from thread pool in real code
        cout << "Processing " << fn1 << " & " << fn2
            << " from " << this_thread::get_id() << '\n';
    };

    for_each(execution::par, vecFilenames1.begin(), vecFilenames1.end(), f);

    return 0;
}

Upvotes: 2

Related Questions