CODEWITHSUNDEEP
apache-cameljms
Ted Herrlich
Ted Herrlich

Reputation: 169

How do I introduce a Delay in Camel to prevent file locking before the files are copied in?

I am using Camel, ActiveMq, and JMS to poll a directory and process any files it finds. The problem with larger files is they start processing before being fully copied into the directory. I has assumed (yes, I know what assume gets you) that the file system would prevent it -- but that doesn't seem to be true. The examples in the Camel docs do not seem to be working. Here is my code from within the configure method of the RouteBuilder:

    from("file://" + env.getProperty("integration.directory.scan.add.eng.jobslist")+"?consumer.initialDelay=100000")
        .doTry()
            .setProperty("servicePath").constant("/job")
            .setProperty("serviceMethod").constant("POST")
            .process("engImportJobsFromFileProcessor")
        .doCatch(Exception.class)
            .to("log:-- Add Job(s) Error -------------------------")
            .choice()
                .when(constant(env.getProperty("eng.mail.enabled.flag.add.jobslist.yn")).isEqualToIgnoreCase("Y"))
                    .setHeader("subject", constant(env.getProperty("integration.mq.topic.add.eng.jobslist.error.email.subject")))
                    .to("direct://email.eng")
                .otherwise()
                    .to("log:-----------------------------------------")
                    .to("log:-- Email for JOBSLIST IS DISABLED")
                    .to("log:-----------------------------------------")
            .end()
        .end()
        .log("Finished loading jobs from file ")
    ;

As you can see, I tried to set an 'initialDelay', I have also tried 'delay' and 'readLock=changed' and nothing made a difference. As soon as the file hits the directory, Camel starts processing. All I am after is a nice simple delay before the file is polled. Any ideas?

Upvotes: 0

Views: 5311

Answers (3)

Ted Herrlich
Ted Herrlich

Reputation: 169

OK, turned out to be a combination of things. First off I test inside of IntelliJ and also outside for several reasons -- one is a security issue with using email within IDEA. Tomcat, outside of IntelliJ was picking up a classes folder in the webapps/ROOT directory, which was overwriting my changes to the uri options. That's what was driving me nuts. That ROOT folder had been there from a deployment error from several months ago. But it wasn't being picked up by IntelliJ even though I was using the same Tomcat instance. That's why it appear that my changes were being ignored.

Upvotes: 1

Anurag Ashok
Anurag Ashok

Reputation: 113

Use combination of the options "readLock=changed" , "readLockCheckInterval=1000" and readLockMinAge=20s (1000 is in milliseconds and the default value, should be changed to higher value is writes are slower i.e the file size changes after a long time, this may happen on certain filesystems, that the file size changes not very frequently while transfer is in process)

The file component documentation @ http://camel.apache.org/file2.html says

for readlock=changed

changed is using file length/modification timestamp to detect whether the file is currently being copied or not. Will at least use 1 sec. to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency.

for readLockCheckInterval=1000

Camel 2.6: Interval in milliseconds for the read-lock, if supported by the read lock. This interval is used for sleeping between attempts to acquire the read lock. For example when using the changed read lock, you can set a higher interval period to cater for slow writes. The default of 1 sec. may be too fast if the producer is very slow writing the file.

for readLockMinAge=20s

Camel 2.15: This option applies only to readLock=change. This option allows you to specify a minimum age a file must be before attempting to acquire the read lock. For example, use readLockMinAge=300s to require that the file is at least 5 minutes old. This can speedup the poll when the file is old enough as it will acquire the read lock immediately.

So in the end your endpoint should look something like

from("file://" + env.getProperty("integration.directory.scan.add.eng.jobslist")+"?consumer.initialDelay=100000&readLock=changed&readLockCheckInterval=1000&readLockMinAge=20s")

Upvotes: 1

Bedla
Bedla

Reputation: 4919

Use option readLockMinAge.

From File2 component documentation:

This option allows you to specify a minimum age a file must be before attempting to acquire the read lock. For example, use readLockMinAge=300s to require that the file is at least 5 minutes old.

For 100s delay could URI look like this:

from("file://" + env.getProperty("integration.directory.scan.add.eng.jobslist")+"?readLock=changed&readLockMinAge=100s")

Upvotes: 3

Related Questions