Camel - Stop route when the consuming directory not exists

Question

I've an SFTP route (in Spring XML), and its from path ends in a daily changing directory (ie. /yyyyMMdd), and everything is working well when autoCreate=true or the directory exists when the route starts. But it is not permitted to me to create the directory if not exists!

When the dir exists, the route get files and terminates itself.

When the dir not exists, the route is polling permanently with a warning (ie. org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: 20160917) and never stops.

How can I avoid this behaviour (eg. convert the warning to an empty message or exception or ...)? I've done experiments with startingDirectoryMustExist, consumer.bridgeErrorHandler and many others without any success.

The simplified route (before start, fill the elmu.sftp.importDir property with the actual date):

    <from
        uri="sftp://{{elmu.sftp.host}}:{{elmu.sftp.port}}{{elmu.sftp.importDir}}?username={{elmu.sftp.userName}}&amp;password={{elmu.sftp.password}}&amp;
        autoCreate=false&amp;preferredAuthentications=password&amp;binary=true&amp;include={{elmu.importMask}}&amp;initialDelay=100&amp;
        noop=true&amp;sortBy=file:name&amp;sendEmptyMessageWhenIdle=true"/>
    <choice>
        <when>
            <simple>${body} != null</simple>
            ... a lot of stuff ...
            <to uri="bean:shutdownRoute" />
        </when>
        <otherwise>
            <to uri="bean:shutdownRoute" />
        </otherwise>
    </choice>

With directoryMustExist=true and startingDirectoryMustExist=true the result is an endless loop (poll) with this warning:

08:30:14,658 WARN  SftpConsumer - Consumer Consumer[sftp://xxx.xxx.xx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&username=xxx] failed polling endpoint: Endpoint[sftp://xxx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&username=xxx]. Will try again at next poll. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: 20160918] 
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: 20160918
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:576)
    at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:564)
    at org.apache.camel.component.file.remote.SftpConsumer.doPollDirectory(SftpConsumer.java:107)
    at org.apache.camel.component.file.remote.SftpConsumer.pollDirectory(SftpConsumer.java:79)
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:131)
    at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:175)
    at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:102)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
    at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:2340)
    at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:342)
    at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:574)
    ... 13 more

It doesn't work with stepwise=false:

11:52:19,210 WARN  SftpConsumer - Consumer Consumer[sftp://xxx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&stepwise=false&username=xxx] failed polling endpoint: Endpoint[sftp://xxx:22/DBHtest/ELMUteszt/Kiadott_adatok/20160918?autoCreate=false&binary=true&directoryMustExist=true&include=%5E.*%24&initialDelay=100&noop=true&password=xxxxxx&preferredAuthentications=password&sendEmptyMessageWhenIdle=true&sortBy=file%3Aname&startingDirectoryMustExist=true&stepwise=false&username=xxx]. Will try again at next poll. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot list directory: DBHtest/ELMUteszt/Kiadott_adatok/20160918] 
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot list directory: DBHtest/ELMUteszt/Kiadott_adatok/20160918

UPDATE (according to @ruffp's answer):

I've tried to set up a custom PollingConsumerPollStrategy, but I cannot stop the route from it. Only the third (commented row) stops the route, but I've several routes and I cannot know the name of the actual route. How can I get it?

    @Override
    public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws Exception {
        consumer.getEndpoint().stop(); // 1
        consumer.stop();  // 2
        consumer.getEndpoint().getCamelContext().stopRoute(route???);  // 3
        return false;
    }

Answer 1

Otto Csatari: To get Route Id of failed route in rollback method use below code snipped

    CamelContext context = endpoint.getCamelContext();
    List<Route> routes = context.getRoutes();
    SftpEndpoint sftpEndpoint = (SftpEndpoint)endpoint;
    Route failedRoute =  routes.stream().filter(route -> ((SftpEndpoint) route.getEndpoint())
            .getConfiguration().getDirectoryName().equals(sftpEndpoint.getConfiguration().getDirectoryName()))
            .findAny().orElse(null);

Answer 2

Finally I've solved it with consumer.exceptionHandler. But it seems, that this option is not in list of the usable options (http://camel.apache.org/file2.html which I've been reading again and again) just mentioned once with an example at the bottom of the huge page. Unfortunately, it was so "hidden" that I missed it up to now.

I set up a new class and implemented the handleExceptions methods:

public class DirNotExistsExHandler implements ExceptionHandler

which get the exceptions and decide what to do. In the context, I did a bean definition:

<bean id="dirNotExistsExHandler" class="hu.dbit.eleo.DirNotExistsExHandler" />

And in the consumer, passed the bean to the handler:

consumer.exceptionHandler=#dirNotExistsExHandler

Many thanks for your help!

Answer 3

In my opinion the best way to do it would be:

1. Enable the option throwExceptionOnConnectFailed on the sftp endpoint like this:

   sftp://{{elmu.sftp.host}}:{{elmu.sftp.port}}{{elmu.sftp.importDir}}?username={{elmu.sftp.userName}}&password={{elmu.sftp.password}}&
   autoCreate=false&preferredAuthentications=password&binary=true&include={{elmu.importMask}}&initialDelay=100&
   noop=true&sortBy=file:name&sendEmptyMessageWhenIdle=true&throwExceptionOnConnectFailed=true
   

   This could help to manage with the exception and to be catch it by the camel routes onException. However I am not sure it is really necessary in your case.

2. Make a special processing for your connection exception

   // 2a) Solution to redirect an empty body to a destination
   onException(GenericFileOperationFailedException.class)
               .handled(true)
               .log(LoggingLevel.INFO, "Source directory not present: send empty body to shutdown route...")
               .setBody(null)
               .to("bean:shutdownRoute");
   

   Or another way:

   // 2b)Solution to just stop the processing without log in warn or error
   onException(GenericFileOperationFailedException.class)
       .handled(true)
       .log(LoggingLevel.INFO, "Source directory not present: stop the process and wait until next time...")
       .stop();
   

UPDATE: I found this post and apparently there is not another way than implementing your own PollingConsumerPollStrategy because the GenericFileOperationFailedException is apparently handled inside the default implementation.

Answer 4

Handle the exception using doCatch block. If no exception is thrown then check file exists using your code and throw exception manually.

http://camel.apache.org/try-catch-finally.html