CODEWITHSUNDEEP
ruby-on-railsrubysidekiq
Amanda Ferrari
Amanda Ferrari

Reputation: 966

How to use sidekiq to avoid coupling of two queues

Right now I'm using Sidekiq to in parallel calculate two things: tags & categories of a blog post. This calculation should happen once the post is saved to the database. So I have:

class Post < ApplicationRecord

  after_commit :extract_tags_and_categories, if: :saved_change_to_content?

  private

  def extract_tags_and_categories
    TagsExtractorWorker(id)
    CategoriesExtractorWorker(id)
  end
end

The problem with that is that I'm using a third party API to help me fetch only the text of the url of a post, and both of them need ithe text of it. So I'm doing:

class TagsExtractorWorker < Worker
  def perform(post_id)
    post = Post.find!(post_id)
    text = ThirdPartyAPI.get(post.url)
    # find out the tags here and save it to post.tags
  end
end

class CategoriesExtractorWorker < Worker
  def perform(post_id)
    post = Post.find!(post_id)
    text = ThirdPartyAPI.get(post.url)
    # find out the categories here and save it to post.categories
  end
end

As you can imagine, this API is not free and there is a limit of API calls (I would have to pay for more in order to get be able to call it more times), and right now I'm doing two calls (on both TagsExtractorWorker and CategoriesExtractorWorker). I would like to be able to call it only once, and use it in both classes. Does sidekiq have something to help me with that?

Also, I can't really use the Post class to this:

class Post < ApplicationRecord

  after_commit :extract_tags_and_categories, if: :saved_change_to_content?

  private

  def extract_tags_and_categories
    text = get_text_somehow()
    TagsExtractorWorker(id, text)
    CategoriesExtractorWorker(id, text)
  end
end

Because I'm calling other things (besides these two classes), and I don't want to wait and create coupling between things.

My question is: does sidekiq has something to help me with that? Perhaps a middleware that would fetch the text and cache it for other calls.

Upvotes: 0

Views: 110

Answers (1)

Jon Wolski
Jon Wolski

Reputation: 2833

You should be able to cache the HTTP response with your HTTP client, if your third-party API responds with adequate HTTP caching headers.

I don't know of anything in Sidekiq that does that for you. Personally, I've used https://github.com/plataformatec/faraday-http-cache (with a cache store of memcache or redis) for this in the past.

Of course, this will be an option for you only if you can add the caching middleware to your ThirdPartyAPI HTTP client.

If that works, then your Sidekiq workers can continue to call through ThirdPartyAPI as they are coded to do now without actually hitting your third party (and counting against your quota).

Upvotes: 1

Related Questions