CODEWITHSUNDEEP
gpucaffeconv-neural-network
user3450049
user3450049

Reputation: 865

how to speed up image loading in caffe?

While training a CNN using caffe, between every iteration printed out, I get between 3 and 5 repetitions of the following warning:

blocking_queue.cpp:50] Waiting for data

I'm using the default caffe settings in that I made an lmdb database using the provided script. The images are 200x200px tifs, about 30-50kB before loading into the lmdb.

How should I diagnose where the slowdown is? I.e. what part of the hardware chain would need to be upgraded, or what I can do with software to speed the processing up. I'm using GeForce GTX 980 GPUs with CUDA Version 7.5.18.

Upvotes: 1

Views: 1144

Answers (1)

Prune
Prune

Reputation: 77885

This is likely a problem with your I/O chain. Do you have an application to monitor I/O capability to the device? Is there slow response at at the same time as "Waiting for data"?

One possible workaround is to increase the batch size: reading in more data at once will keep your model busy longer and give the I/O routines more time to fetch the next data block.

Upvotes: 2

Related Questions