CODEWITHSUNDEEP
spring-integrationspring-integration-sftp
Pradip Sarkar
Pradip Sarkar

Reputation: 33

Spring SFTP polling multiple directory using Outbound Message Gateway and IntegrationFlow

I want to get all file under a particular remote directory in a periodic manner. I am able to get the file under that directory only once after application startup. Not sure why the Poller is not working. This is registered in a spring boot project and the version is 2.2.1

@InboundChannelAdapter(value = "sftpReportChannel",
            poller = @Poller(fixedDelay = "5000"))
    public String filesForGET(){
        return "/etl/biq/autoscore/output/report-data/";
    }


    @Bean
    public IntegrationFlow sftpGetFlow(SessionFactory<ChannelSftp.LsEntry> csf) {
        return IntegrationFlows.from("sftpReportChannel")
                .handle(Sftp.outboundGateway(csf,
                                AbstractRemoteFileOutboundGateway.Command.LS, "payload")
                        .options(AbstractRemoteFileOutboundGateway.Option.RECURSIVE,  AbstractRemoteFileOutboundGateway.Option.NAME_ONLY)
                        //Persistent file list filter using the server's file timestamp to detect if we've already 'seen' this file.
                        .filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data")))
                .split()
                .log(message -> "file path -> "+message.getPayload())
                .handle(Sftp.outboundGateway(csf, AbstractRemoteFileOutboundGateway.Command.GET, "'/etl/biq/autoscore/output/report-data/' + payload")
                        .options(AbstractRemoteFileOutboundGateway.Option.STREAM))
                .handle(new ReportHandler()) //get the payload and create email content and send eamil to recipients
                .get();
    }

Upvotes: 0

Views: 442

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121272

The .filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data"))) makes it working the way that it doesn't pick up the same file again and again on the subsequent poll activities.

Make sure you add new files in that remote dir at runtime or modify already processed file. The SftpPersistentAcceptOnceFileListFilter logic relies on the mtime property of the LsEntry to determine that the file has been changed therefore it is good for processing again.

Upvotes: 1

Related Questions