CODEWITHSUNDEEP
file-ioiterator
Mao
Mao

Reputation: 77

Find the best solution for read file and call web service

I have a text file which has about 100,000 records of identifier.

I must read all of record, each record i do request to web service and receive the result from web service, the result i write to another file.

I'm confuse between two solution: - Read identifier file to a list of identifier, iterate this list, call web service, .... - Read identifier line on each line, call web service, .....

Do you think what solution will be better ? program will do faster ?

Thanks for all.

Upvotes: 0

Views: 195

Answers (1)

Coral Doe
Coral Doe

Reputation: 1941

As Dukeling says, using different threads to read the file, send requests and write to file can increase the speed of the program, rather the one thread solutions you propose.

I recommend that you would start using asynchronous calls to your web service. You make the call, but don't wait for a response (you handle the responses in the callback). When you make a lot of calls to the web service in parallel (as you want speed), this frees up some I/O threads on your hosting machine and can improve the rate/time of processed requests sometimes. Then you can have a thread that reads from the file, starts the asynchronous call and repeats. On the callback function you implement the writing to file. You should at this level implement a logic that insures that your responses are written in the right order.

On the other hand, calling the web service for each record may be too chatty. I would suggest an implementation similar to pagging: loading a certain amount of records, sending them to operation and receiving the responses in bulk. You should take care of not failing the whole package for one recors, have a logic for resending only a part of the tasks and so on.

Upvotes: 1

Related Questions