CODEWITHSUNDEEP
spring-integration
max3d
max3d

Reputation: 1507

How to handle error during file transformation

When a large file is uploaded to a polled directory, and is attempted to be unzipped, an error takes place, because the file is not yet complete.

How to make a retry of polling after some time?

@Bean
@InboundChannelAdapter(value = "inputChannel", poller = @Poller(fixedRate = "1500"))
public FileReadingMessageSource poll() {
    FileReadingMessageSource source = new FileReadingMessageSource();
    source.setScanEachPoll(true);
    source.setDirectory(new File(pathConfig.getIncomingDirPath()));
    source.setUseWatchService(true);
    source.setFilter(new SimplePatternFileListFilter("*.zip"));
    return source;
}

@Transformer(inputChannel = "inputChannel", outputChannel = "unzipChannel")
public Message convert(Message<File> fileMessage) {
    UnZipTransformer unzipTransformer = new UnZipTransformer();
    unzipTransformer.setZipResultType(ZipResultType.FILE);
    unzipTransformer.setWorkDirectory(new File(pathConfig.getWorkDirPath()));
    unzipTransformer.setDeleteFiles(false);
    unzipTransformer.afterPropertiesSet();
    File file = fileMessage.getPayload();
    return unzipTransformer.transform(fileMessage);
}

EDIT - i can't get LastModifiedFileListFilter to work with large files. It works fine with small ones; and when I upload a large one, it does not react any more.

EDIT2 Thanks to advice from Artem and Gary, here is the solution that worked for me.

@Bean
@InboundChannelAdapter(value = "inputChannel", poller = @Poller(fixedDelay = "1500"))
public FileReadingMessageSource poll() {
    FileReadingMessageSource source = new FileReadingMessageSource();
    source.setScanEachPoll(true);
    source.setDirectory(new File(pathConfig.getIncomingDirPath()));
    source.setUseWatchService(true);
    FileListFilter simplePatternFileListFilter = new SimplePatternFileListFilter("*.zip");
    source.setFilter(new ChainFileListFilter<>().addFilter(simplePatternFileListFilter));
    return source;
}

@Transformer(inputChannel = "inputChannel", outputChannel = "unzipChannel", 
  adviceChain = "retryOnIncompleteData")
public Message convertZip(Message<File> fileMessage) {
    UnZipTransformer unzipTransformer = new UnZipTransformer();
    unzipTransformer.setZipResultType(ZipResultType.FILE);
    unzipTransformer.setWorkDirectory(new File(pathConfig.getWorkDirPath()));
    unzipTransformer.setDeleteFiles(false);
    unzipTransformer.afterPropertiesSet();
    return unzipTransformer.transform(fileMessage);
}

@Bean
public Advice retryOnIncompleteData() {
    RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
    RetryTemplate template = createRetryTemplate();
    advice.setRetryTemplate(template);
    return advice;
}

private RetryTemplate createRetryTemplate() {
    RetryTemplate template = new RetryTemplate();
    SimpleRetryPolicy policy = new SimpleRetryPolicy();
    policy.setMaxAttempts(15);
    template.setRetryPolicy(policy);
    FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
    backOffPolicy.setBackOffPeriod(25000l);
    template.setBackOffPolicy(backOffPolicy);
    return template;
}

Upvotes: 1

Views: 253

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121282

Well, actually since you use only SimplePatternFileListFilter, any your files are going to be retried on each poll interval.

To avoid re-fetching the same file over and over (if you don't delete it in the end of the process), we recommend to use AcceptOnceFileListFilter as a part of the CompositeFileListFilter.

In case of error like yours you can use ExpressionEvaluatingRequestHandlerAdvice to call onFailureExpression to ResettableFileListFilter.remove() implementation of the AcceptOnceFileListFilter.

On the other hand instead of ExpressionEvaluatingRequestHandlerAdvice you can consider to use RequestHandlerRetryAdvice to retry an unzipping process: you don't need to re-fetch file from the localsystem.

These AOP advices you should apply to the @Transformer.

See more info in the Reference Manual.

BTW, I would say fixedRate is not good for large files. Especially when you have errors like this. The fixedDelay would be better. See their JavaDocs for more information.

Upvotes: 1

Related Questions