CODEWITHSUNDEEP
ruby-on-railsrubymultithreadingparallel-processing
Alok Mysore
Alok Mysore

Reputation: 616

Ordered insertion of data with parallel processing in ruby

Here's my Ruby code

The method getLikes() fetches the Facebook likes in a hash format of the user with token t

def multi
    token = ["Facebook token 1","Facebook token 2","Facebook token 3",...]
    @data = []
    @threads = []

    token.each do |t|
        @threads << Thread.new{@data << getLikes(t)}
    end

    @threads.each do |th|
        th.join
    end


    render json: @data
end

The problem is that the data that @data's contents are not in the same order because of the parralization.

To fix this, I modified the first loop to

i = 0
token.each do |t|
    @threads << Thread.new{@data[i] = getLikes(t)}
    i = i + 1
end

But then the program does not wait for all the threads to finish. I get a few null values in the @data array.

What would be a good way to go about this?

Thanks

Upvotes: 4

Views: 214

Answers (2)

Janitrix
Janitrix

Reputation: 203

The issue is that your code is not thread-safe as it uses a shared-variable among threads without using mutexes. Hashes are not thread-safe in Ruby.

The solution is to return simple values in your threads, and aggregate the results in your main code. To keep the order, simply return the token along with the value:

def multi
  token = ["Facebook token 1","Facebook token 2","Facebook token 3",...]
  @threads = []

  token.each do |t|
    @threads << Thread.new{[t, getLikes(t)]}
  end

  data = Hash[@threads.map(&:value)] # this will block the main thread

  sorted_data = []
  token.each do |t|
    sorted_data << data[t]
  end

  render json: sorted_data
end

Upvotes: 3

Uri Agassi
Uri Agassi

Reputation: 37419

Try:

token.each_with_index do |t, 1|
  @threads << Thread.new{@data[i] = getLikes(t)}
end

The problem with your i = i + 1 solution is that the value of i in the Thread block is taken when the block is run, and not when it is declared. This means that you might get one or two items in some spot in the middle of the array, but most of your threads will take the last value of i, and override each other at the last position of the array...

Upvotes: 0

Related Questions