CODEWITHSUNDEEP
javadatabaseweb-servicesjsp
Shahriar
Shahriar

Reputation: 824

migrating large data between systems

I need a suggestion to implement a functionality for my application. My application is a legacy JSP , JDBC application installed in client environments used by internal employees of the client. I have multiple client and each client have their own environment of my software installed in their system.

A a requirement, In one module we need to have an interface to their existing external application modules to synchronize some data between their system and our system (eg. user info, office info).

As first step, Right now we have some module where the client can upload CSV,EXCEL files and upload to our system's temporary tables.

Then in second step, they use another module where the trigger the instruction to get data from the temporary table which is processed according to our data format and then inserted to our mail tables to be used into our system.

Now, we are planning to provide them some API where they an directly insert data from their system/module to the temporary tables instead of uploading the CSV,EXCEL files. So that it will not be our responsibility to format data and load into temporary tables.

They will access our API and executes specific methods.

My question is how this can be done? We already have a web service where we expose a small amount of our system just to modify/insert data into specific table. But I am afraid if inserting / synchronizing 40k-50k of rows with web service POST call will be a good idea.

Final point is how can we develop a API that can be used from any client (PHP,.net, java) which I think web service is the solutions.

I hope I did explain clearly. Help appreciated.

Upvotes: 0

Views: 90

Answers (1)

Alejandro Diaz
Alejandro Diaz

Reputation: 431

Yes, you should provide a bulk capability to your web service. Doing 50k or more calls it's not efficient. So, since you already defined a format for a file and it looks that people is familiar with it, you can write a WS that receives the file as multipart. See http://cxf.apache.org/docs/jax-rs-multiparts.html

Additionally, I'd suggest to do this call async. You can return a "JobId" for future reference.

NOTE1: If you think you an provide a new format, use Json. You can provide some RESTFul API. SOAP with XML might have some overhead. NOTE2: Since you are now using HTTP. Be aware of the security risks.

Upvotes: 2

Related Questions