CODEWITHSUNDEEP
apache-camelapache-commons-netcamel-ftp
Camel User
Camel User

Reputation: 21

Apache Camel FTP- Consumption from FTP slows down on increasing the number of files, while the total size of files remain same

I am new to camel. I am trying to consume from secure ftp server to my local file folder but the consumption speed is really very slow.

Please suggest the parameters which might help in tuning the performance .

My route is as follows –

ftps://[email protected]:21/files/100kbFiles?connectTimeout=60000&ftpClient.keyStore.file=.%2Fsrc%2Ftest%2Fresources%2Fwip.jks&ftpClient.keyStore.keyPassword=xxxxxx&ftpClient.keyStore.password=xxxxxx&localWorkDirectory=%2Ftmp&maximumReconnectAttempts=3&move=.done&passiveMode=true&password=xxxxxx&securityProtocol=SSL&soTimeout=60000]

To Endpoint - file://target/toFolder

It takes almost 16 minutes to consume 200 files of size 200KB each.

I also tried to use Stream caching –

context.getStreamCachingStrategy().setSpoolDirectory("/tmp/cachedir");
                           context.getStreamCachingStrategy().setSpoolThreshold(1024 * 102);
                           context.getStreamCachingStrategy().setBufferSize(1024 * 1024 );

from Endpoint -> 

ftps://[email protected]:21/files/100kbFiles?connectTimeout=60000&delay=3000&eagerMaxMessagesPerPoll=true&ftpClient.keyStore.file=.%2Fsrc%2Ftest%2Fresources%2Fwip.jks&ftpClient.keyStore.keyPassword=xxxxxx&ftpClient.keyStore.password=xxxxxx&maxMessagesPerPoll=200&maximumReconnectAttempts=3&move=.done&passiveMode=true&password=xxxxxx&securityProtocol=SSL&soTimeout=60000]

To Endpoint -> file://target/toFolder it is also taking almost 15 minutes to consume 200 files of size 100kB each….

While when I use the same route for consuming 20 files of size 1 MB files each, it is only taking 1 minute and 30 Seconds.

I am unable to understand why camel is slowing down when the batch size increases.

Please suggest, what parameter I am missing which will improve the performance of FTPS.

I am using camel 2.15.2, Commons net API 3.3.

Thanks,

Upvotes: 2

Views: 1483

Answers (1)

Souciance Eqdam Rashti
Souciance Eqdam Rashti

Reputation: 3191

One issue that comes to mind is that on the camel-ftp component site http://people.apache.org/~dkulp/camel/ftp2.html, it says

FTP Consumer does not support concurrency The FTP consumer (with the same endpoint) does not support concurrency (the backing FTP client is not thread safe). You can use multiple FTP consumers to poll from different endpoints. It is only a single endpoint that does not support concurrent consumers.

The FTP producer does not have this issue, it supports concurrency.

To check, why not use the 100KB size files, but first try with 1 file, then 10, then 25, then 50 and then 100, 125, 150 and 200 and see after what point it starts taking too long time.

After that I would look into using batch consumer to see if it can help speed things up buy polling many files at once. You can find info here: http://camel.apache.org/batch-consumer.html together with camel-ftp and How Camel 2.11 batch aggregation works with separate route?

Upvotes: 0

Related Questions