CODEWITHSUNDEEP
rubyruby-on-rails-3queueresque
Kostas
Kostas

Reputation: 8595

Consequent attribute calculations with a queuing system

For all of the following assume these:

We have 3 models:

When we update the product (let's say we disable it) we need to have some things happen to the relevant sku and category. The same is true for when a sku is updated.

The proper way of achieving this is have an after_save on each model that triggers the other models' update events.

example:

products.each(&:disable!)
# after_save triggers self.sku.products_updated
#   and self.category.products_updated (self is product)

Now if we have 5000 products we are in for a treat. The same category might get updated hundreds of times and hog the database while doing so.

We also have a nice queueing system, so the more realisting way of updating products would be products.each(&:queue_disable!) which would simply toss 5000 new tasks to the working queue. The problem of 5000 category updates still exists though.

Is there a way to avoid all those updates on the db?

How can we concatenate all the category.products_updated for each category in the queue?

Upvotes: 5

Views: 156

Answers (2)

tee
tee

Reputation: 4429

You can ensure a single category update for all the products by using a couple Resque plugins: Resque Unique Job and Resque Scheduler.

Delay the execution of the job to update the category slightly (however long it takes to typically call all the product updates) and ensure each job is unique by including the Unique Job module. Unique Job uses the paramaters of the job, so if you try to queue 2 jobs with category_id 123, it ignores the 2nd one since the job is already queued.

class Product
  after_save :queue_category_update

  def queue_category_update
    Resque.enqueue_at(1.minute.from_now, Jobs::UpdateCategory, category.id) if need_to_update_category?
  end
end

module Jobs
  module UpdateCategory
    include Resque::Plugins::UniqueJob

    def self.perform(category_id)
      category = Category.find_by_id(category_id)
      category.update_some_stuff if category
    end
  end
end

Upvotes: 2

tee
tee

Reputation: 4429

Do the dependent updates in single SQL calls. #update_all will update many records at once. For example,

In an after_update callback, update all the dependent column values:

class Category
  after_update :update_dependent_products

  def update_dependent_products
    products.update_all(disabled: disabled?) if disabled_changed?
  end
end

If that's too slow, move it into a resque job:

class Category
  after_update :queue_update_dependent_products

  def update_dependent_products
    products.update_all(disabled: disabled?) if disabled_changed?
  end      

  def queue_update_dependent_products
    Resque.enqueue(Jobs::UpdateCategoryDependencies, self.id) if disabled_changed?
  end
end

class Jobs::UpdateCategoryDependencies
  def self.perform(category_id)
    category = Category.find_by_id(category_id)
    category.update_dependent_products if category
  end
end

Do similar things for the other model callbacks.

Upvotes: 0

Related Questions