CODEWITHSUNDEEP
elixirphoenix-framework
TheStoneFox
TheStoneFox

Reputation: 3067

Pushing operations to a message queue for processing when able to

My title isn't great because I'm finding it hard to explain what I want in a small snippet...soo I'll try in depth!

I'm writing an Elixir app using the Phoenix web framework.

What I want to do is this:

The process does not need to return back to the web route (i.e. its Async)

I know how I would handle this in other languages by using a message queue (Beanstalk, RabbitMQ, sidekiq) and have something that hands the "job" off to a worker and then write a worker to handle the jobs on the queue.

But....I know in Elixir it has queues by default as everything gets pushed into a "mailbox" (saw it in a youtube video :p)

But I cant find any decent code examples that show how to have a controller (in phoenix) spawn an async process that will be processed off the message queue.

Does anyone know of any decent resources that show what I'm looking for, or does anyone have any code snippets that would give me a basic description of how to do it.

Thanks :)

EDIT:

As mentioned in the comments, I've created a basic tutorial for how to get Poolboy up and running in an Elixir app. I hope it is helpful (and correct!) for people :)

https://github.com/thestonefox/elixir_poolboy_example

Upvotes: 8

Views: 2628

Answers (2)

José Valim
José Valim

Reputation: 51429

If the jobs are "important", you still want to use some storage to persist them. Otherwise, if the node goes down for any reason, you lose all jobs that are in the system. If this is the case, you can use something like RabbitMQ and create a bunch of workers in your application:

children = [
  worker(QueueConsumer, [], id: :consumer_1),
  worker(QueueConsumer, [], id: :consumer_2),
  worker(QueueConsumer, [], id: :consumer_3),
  worker(QueueConsumer, [], id: :consumer_4)
]

The consumers would get the next message from RabbitMQ and process them as they come.

If you don't care about persisting them or if you want to regulate them based on other criteria like CPU usage, then I recommend you to look into the jobs library (in Erlang): https://github.com/uwiger/jobs

Upvotes: 9

Fred the Magic Wonder Dog
Fred the Magic Wonder Dog

Reputation: 9109

Sounds like you are describing Poolboy.

http://hashnuke.com/2013/10/03/managing-processes-with-poolboy-in-elixir.html

Upvotes: 5

Related Questions