CODEWITHSUNDEEP
filewso2wso2-streaming-integrator
Fateme Shahrabi
Fateme Shahrabi

Reputation: 103

WSO2 streaming integrator - read large files (one million of records) and write to another file

I want to read one million records from a CSV file and write to another CSV file with WSO2 streaming integrator. The problem is the slow speed of reading and writing data, almost 10 per second. WSO2 says "Read a 100GB file with 10 million lines in just 20 mins" but I found no sample or guide to do that. can anyone help me?

my simple task flow for reading and writing data

Upvotes: 0

Views: 244

Answers (1)

Fateme Shahrabi
Fateme Shahrabi

Reputation: 103

Finally I found some example here https://ei.docs.wso2.com/en/7.2.0/streaming-integrator/ref/performance-analysis-results/ and with these example my speed is ok.

batch.size.max and buffer.size are useful parameters here.

Upvotes: 2

Related Questions