CODEWITHSUNDEEP
architecturemicroservicesapache-nifidistributed-system
Subham Rakshit
Subham Rakshit

Reputation: 13

Microservice architecture

We have multiple separate applications written in Java,C#,nodeJS and python. All these application share a common property - they pull data from some source based on schedule per customer basis using REST API and store it in CSV file, later import data from CSV file to different SQL database using stored procs. Each application is used to integrate data from different third party service.

For example - app A fetching data from source A' and app B fetching data from source B'

I'm thinking of replacing these multiple separate applications by writing one multi-tenant single application which can handle pulling data from different sources. All these separate small applications will be configured as a custom job written in Java. For example - REST API authentication, pre-processing of data before creating CSV etc.

So, I want to write a job (Java file) that fetches data from source A' and another job that fetches data from source B'. The main application will execute this custom job. The common functionality such as job scheduling, logging etc will be supported by the main application.

Later I intend to use nifi to handle data import from CSV to SQL database.

Will this be a good approach? I'm planning to write this application in Java.

Reason behind the solution

If I need to update one job I need to deploy the whole application. How to get around this process? Is there any way I can deploy just the job and not the whole application?

What would be the good way to architect this solution?

Upvotes: 0

Views: 847

Answers (1)

Andy
Andy

Reputation: 14194

I think if you are able to adapt your nomenclature/approach a bit, you will find that you can use Apache NiFi to perform almost all of the tasks in your system. While NiFi is designed to handle streaming data rather than batching/jobs, you can use CRON timing to schedule task execution (the initial REST API invocation and data retrieval) and then have the flowfiles sent to follow-on processors. With this model, you can ingest data, treat the data as discrete records and convert easily between various formats like XML, JSON, CSV, Avro, etc., filter, route, transform, and query the data, and then route it to the destination SQL database. All of these operations will have live monitoring, provenance records, auditing, replayability, logging, fine-grain user access controls, etc. This will also have the added benefit that you do not need to write any code to handle these activities -- you simply drag and drop the components on the flow. You can make modifications to any part of the flow without influencing unrelated operations, so there is no interruption to the data flow.

Upvotes: 1

Related Questions