CODEWITHSUNDEEP
multithreadingluatorch
mattdns
mattdns

Reputation: 894

Torch - Multithreading to load tensors into a queue for training purposes

I would like to use the library threads (or perhaps parallel) for loading/preprocessing data into a queue but I am not entirely sure how it works. In summary;

  1. Load data (tensors), pre-process tensors (this takes time, hence why I am here) and put them in a queue. I would like to have as many threads as possible doing this so that the model is not waiting or not waiting for long.
  2. For the tensor at the top of the queue, extract it and forward it through the model and remove it from the queue.

I don't really understand the example in https://github.com/torch/threads enough. A hint or example as to where I would load data into the queue and train would be great.

EDIT 14/03/2016

In this example "https://github.com/torch/threads/blob/master/test/test-low-level.lua" using a low level thread, does anyone know how I can extract data from these threads into the main thread?

Upvotes: 1

Views: 2078

Answers (2)

Octavian Ganea
Octavian Ganea

Reputation: 61

If Soumith's examples in the previous answer are not very easy to use, I suggest you build your own pipeline from scratch. I provide here an example of two synchronized threads : one for writing data and one for reading data:

local t = require 'threads'
t.Threads.serialization('threads.sharedserialize')
local tds = require 'tds'
local dict = tds.Hash()  -- only local variables work here, and only tables or tds.Hash()
dict[1] = torch.zeros(4)

local m1 = t.Mutex()
local m2 = t.Mutex()
local m1id  = m1:id()
local m2id  = m2:id()

m1:lock()

local pool = t.Threads(
  1,
  function(threadIdx)
  end
)

pool:addjob(
  function()
    local t = require 'threads'
    local m1 = t.Mutex(m1id)
    local m2 = t.Mutex(m2id)

    while true do
      m2:lock()
      dict[1] = torch.randn(4)
      m1:unlock()

      print ('W ===> ')
      print(dict[1])
      collectgarbage()
      collectgarbage()      
    end

    return __threadid
  end,
  function(id)
  end
)

-- Code executing on master:
local a = 1
while true do
  m1:lock()
  a = dict[1]
  m2:unlock()

  print('R --> ')
  print(a)
end

Upvotes: 0

smhx
smhx

Reputation: 2266

Look at this multi-threaded data provider: https://github.com/soumith/dcgan.torch/blob/master/data/data.lua

It runs this file in the thread: https://github.com/soumith/dcgan.torch/blob/master/data/data.lua#L18

by calling it here: https://github.com/soumith/dcgan.torch/blob/master/data/data.lua#L30-L43

And afterwards, if you want to queue a job into the thread, you provide two functions: https://github.com/soumith/dcgan.torch/blob/master/data/data.lua#L84 The first one runs inside the thread, and the second one runs in the main thread after the first one completes.

Hopefully that makes it a bit more clear.

Upvotes: 3

Related Questions