CODEWITHSUNDEEP
erlangrabbitmqetl
vinnylinux
vinnylinux

Reputation: 7024

Extract, transform, load within Rabbit?

One of the things that i do pretty often is transforming SQL data into cache and document-based stores, for performance reasons. I don't want my frontend applications hitting my database, so i have high-speed cache solutions, as well efficient Solr and other solutions.

I use RabbitMQ as the central communication hub to achieve this ETL flow, which looks like this: Backend application sends a message to Rabbit with the new data, or changes made into existing data. I then have a node.js script which consumes the queue, makes small batches of data and populates all the necessary systems: Redis, Mongo, Solr, etc.

However, i'm wondering if there's a better way of doing this. Maybe Rabbit has some kind of scripting support to create erlang logic for queues?

Upvotes: 3

Views: 2327

Answers (1)

Derick Bailey
Derick Bailey

Reputation: 72868

However, i'm wondering if there's a better way of doing this. Maybe Rabbit has some kind of scripting support to create erlang logic for queues?

it doesn't. it's just a message queueing system.

personally, I think your current design sounds good.

The only thing I would wonder, is whether or not each of your target systems has a queue of it's own. That way, any one of them can go down and not affect the others.

I would probably do something like this:

  • back-end produces data message and sends through RMQ
  • RMQ is configured with a fanout exchange, and has one bound queue per target system
  • each system receives the message in it's own queue

otherwise, what you have sounds about right to me!

Upvotes: 3

Related Questions