CODEWITHSUNDEEP
javaspring-integrationspring-integration-dsl
Makky
Makky

Reputation: 17471

SpringIntegration Poller copying file

I have setup File poller with task executor 

ExecutorService executorService = Executors.newFixedThreadPool(10);

            LOG.info("Setting up the poller for directory {} ", finalDirectory);
            StandardIntegrationFlow standardIntegrationFlow = IntegrationFlows.from(new CustomFileReadingSource(finalDirectory),
                    c -> c.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS, 5)
                            .taskExecutor(executorService)
                            .maxMessagesPerPoll(10)
                            .advice(new LoggerSourceAdvisor(finalDirectory))
                    ))


                    //move file to processing first processing                    
                    .transform(new FileMoveTransformer("C:/processing", true))
                    .channel("fileRouter")
                    .get();

I am using AcceptOnceFilter based using lastUpdated .

Its all working fine the issue I had was the directory I poll the file come from external system. The file was still being copied and the application picked up and processed partial data.

How can I make sure the file is completely written before I process.

I cannot change the external system as they just copy the files to the folder. They will not change their system.

Upvotes: 2

Views: 70

Answers (2)

Gary Russell
Gary Russell

Reputation: 174829

I cannot change the external system

How can I make sure the file is completely written before I process.

You can't (reliably).

We provide the LastModifiedFileListFilter; you could also monitor the file size, but either way, this is a brittle technique because a network glitch could cause a pause in the file transfer. Also, it depends on the OS (and remote access mechanism) to update these properties while the file is being copied, which might not always be the case.

Upvotes: 1

Artem Bilan
Artem Bilan

Reputation: 121580

Please, combine your AcceptOnceFileListFilter with the LastModifiedFileListFilter to let those files to live in the directory for some time before your start reading them. So, they might be written fully before you pick them up.

See Docs for more info: https://docs.spring.io/spring-integration/reference/html/files.html#file-reading

Upvotes: 1

Related Questions