CODEWITHSUNDEEP
c++multithreading
WhoAmI
WhoAmI

Reputation: 643

Split threads on several pools

I have some code like this:

void doSth () {// code ...}

vector<string> files_1; // filenames .txt to get text from
vector<string> files_2; // filenames .txt to get text from
// sizes of files_1 and files_2 are ==

mutex m;
// lambda for for_each
auto gogogo= [&](const string& fn1) {
    // Comupute other element via pointer arthimetics
    // Works only with vector, for, say, deque, use container of struct
    const string& fn2 = files_2[&fn1 - files_1.data()];
    
    // do whatever I need in this doSth function
    doSth(fn1, fn2);

    // avoid doing any real work under the lock to benefit from paralleling
    lock_guard<mutex> guard(m);
};

// start threads
for_each(execution::par, files_1.begin(), files_1.end(), gogogo);

Now for_each statement execute lambda so many times as files_1.size() have. If it have 40 it will execute lambda in for_each 40 times. I want files_1 & files_2 to be executed in for_each statement 8 by 8 (every next 8 should be called only after previous 8 executed, not everything for once). How could I do that? Do I need to free memory after every 8 ? (RAM is a question if files_1.size() size is big, thats why I want to do it in the order 8 by 8 ). Ask me if you have any questions.

I use mingw64, g++ 9.2, visual studio 2017 c++17, windows 10 os

Now lambda do this :
filename1.txt
filename2.txt
filename3.txt
filename4.txt
filename5.txt
filename6.txt
filename7.txt
filename8.txt
filename9.txt
filename10.txt
filename11.txt
filename12.txt
filename13.txt
filename14.txt
filename15.txt
filename16.txt
filename17.txt

I want it to be executed like this:

filename1.txt
filename2.txt
filename3.txt
filename4.txt
filename5.txt
filename6.txt
filename7.txt
filename8.txt
// Previous 8 files executed successfully, next 8
filename9.txt
filename10.txt
filename11.txt
filename12.txt
filename13.txt
filename14.txt
filename15.txt
filename16.txt
// prev 8 executed, next
filename17.txt
// ...

Upvotes: 0

Views: 50

Answers (1)

G. Sliepen
G. Sliepen

Reputation: 7983

You have to put for_each() inside a loop, and give it iterators that are 8 elements apart. Here is a possible solution:

#include <iterator>

...

for(auto i = files_1.begin(); i != files_1.end();) {
    auto start = i;
    auto end = std::advance(i, std::min(8, std::distance(i, files_1.end()));

    for_each(execution::par, start, end, gogogo);

    i = end;
}

You can write this more concise by using for_each_n() and the fact that it returns an iterator to the element after the range it just processed:

for(auto i = files_1.begin(); i != files_1.end();) {
    auto count = std::min(8, std::distance(i, files_1.end()));
    i = for_each_n(execution::par, i, count, gogogo);
}

Upvotes: 1

Related Questions