CODEWITHSUNDEEP
architecturequeuerabbitmq
bischoffingston
bischoffingston

Reputation: 657

Transferring files to a message broker

We are setting up a database with a front end program that parses csv files and dumps them into the DB. Files are coming from many locations at the same time and need to be persisted if anything goes wrong.

Is a message broker like rabbitmq a good application for this? I need a queue of some sort that can be read from the back end.

Upvotes: 2

Views: 1650

Answers (1)

Alex G.
Alex G.

Reputation: 949

Is a message broker like rabbitmq a good application for this?

Yes, but I wouldn't recommend you to send large files to RabbitMQ especially if you'd like to use persistent messages. If this is the case, I'd save a file to a storage (cloud, cache, etc) and send its name/path in a message. A worker process would pick up a message, then download a file, parse it and save its contents to database. This way you can distribute files among worker processes and reduce load on broker. It would allow you to scale easily.

Upvotes: 5

Related Questions