CODEWITHSUNDEEP
springspring-integrationsftp
Varun Maurya
Varun Maurya

Reputation: 337

Transfer files on different servers (sftp) using Spring Integration

I have a requirement to build an application in SI which reads an input directory which may consist 1000s of file and copy them to remote servers, say 10 servers where a processor instance will pick them up for processing.The movement of file should be on round-robin fashion so that there is not additional burden on any server while processing them. To elaborate a little more - lets say we have 10 files in input directory then application should copy file 1 on server1, file2 on server2 . . . .... file 10 on server 10.

Sequence doesn't matter what matters is that every server should have equal load.I am fairly new to Spring Integration but I found a sample to do sftp of file using SI

https://github.com/spring-projects/spring-integration-samples/tree/master/basic/sftp

but i am not sure how can I configure it for multiple servers and to have an algo to move files in round-robin fashion.

I will appreciate any tips or suggestion .

I am able to do sftp using below config.

<context:property-placeholder location="classpath:app.properties" />

<int-file:inbound-channel-adapter id="ReaderChannel"
    directory="file:${input.file.dir}" filename-pattern="*.*"
    prevent-duplicates="true" ignore-hidden="true" auto-startup="true">
    <int:poller id="poller" fixed-rate="1" task-executor="myTaskExecutor" />
</int-file:inbound-channel-adapter>

<int-task:executor id="myTaskExecutor"  pool-size="${file.concurrentFilesNum}" queue-capacity="0"   rejection-policy="CALLER_RUNS" />

<int-sftp:outbound-channel-adapter  id="sftpOutboundAdapter" session-factory="sftpSessionFactory" channel="ReaderChannel"
    charset="UTF-8" remote-directory="${output.file.dir}" auto-startup="true">
    <int-sftp:request-handler-advice-chain>
        <int:retry-advice />
    </int-sftp:request-handler-advice-chain>
</int-sftp:outbound-channel-adapter>

<beans:bean id="sftpSessionFactory"     class="org.springframework.integration.file.remote.session.CachingSessionFactory">
    <beans:constructor-arg ref="defaultSftpSessionFactory" />
</beans:bean>

<beans:bean id="defaultSftpSessionFactory"
    class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
    <beans:property name="host" value="${sftp.host}" />
    <beans:property name="privateKey" value="${sftp.private.keyfile}" />
    <beans:property name="privateKeyPassphrase" value="${sftp.private.passphrase}" />
    <beans:property name="port" value="${sftp.serverPort}" />
    <beans:property name="user" value="${sftp.username}" />
    <beans:property name="allowUnknownKeys" value="true" />
</beans:bean>

Upvotes: 1

Views: 2366

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121177

The round-robin is hidden in the DirectChannel with UnicastingDispatcher on the RoundRobinLoadBalancingStrategy.

So, when you have several subscriber to the same DirectChannel, the message will be dispatched to them in the round-robin.

What you need for your use-case is just configure 10 <int-sftp:outbound-channel-adapter> for each your remote server. And use the same simple <channel> definition for their channel attribute.

The <int-file:inbound-channel-adapter> should always send its message to that shared channel with default round-robin strategy.

Upvotes: 1

Related Questions