CODEWITHSUNDEEP
c++multithreadingstxxl
Roel
Roel

Reputation: 19642

stxxl exceeding OS limit of nr of threads

STXXL 1.4.1 on Windows 10 / VS2015.

I have some code in which I essentially use this:

typedef stxxl::vector<RC, 4, stxxl::lru_pager<8>, 2 * 1024> TVector;
typedef stxxl::deque<RC, TVector> CellIndexList;

This is to lower the block size from the default of 2*1024*1024; I had to do this because I have

std::vector<CellIndexList> list(400);

And since the default stxxl::vector allocates a buffer of ~60MB, I had 60*400 MB in my vector, which crashed with bad_alloc (32 bits) - so that's how I came to my current setup.

With this as background, what I'm now running into is that my code crashes with the following stxxl diagnostics:

[STXXL-ERRMSG] Exception thrown in ~vector()

And when I add some additional diagnostics to stxxl::vector::~vector(), I get the following:

Dynamic exception type: class std::system_error
std::exception::what: resource unavailable try again: resource unavailable try again

So, the flush() in the stxxl::vector destructor fails with this exception; from googling it seems that this exception means that the maximum number of threads has been reached.

Now I'm struggling to understand why there would be extra threads created when doing the flush(). I have looked around the stxxl source code and I've found some mentions of different types of disk queues, and I have also tried to enable STXXL_HACK_SINGLE_IO_THREAD in defines.h but all of that is just trying random things as I'm lacking an overview of how many threads stxxl creates for which purposes and under what circumstances.

I've also tried making a standalone problem that exhibits this problem but failed; I can easily do

std::vector<CellIndexList> d(10000);

and fill them with lots of data without issue. So there must be something specific about the way I'm using things in my real code that is causing this threading problem. It's just too large to start chopping things out until I can make the problem go away.

So, my question: is there any place in stxxl where potentially large numbers of threads are being created? Is there anything else I can try to find the problem here?

Upvotes: 0

Views: 121

Answers (0)

Related Questions