CODEWITHSUNDEEP
apache-camel
Fede E.
Fede E.

Reputation: 1918

Locking file while appending data on Camel

I am writing 2 routes to process a files in a directory, those files could have any name, but I need 2 routes as I need some complex processing.

First route:

<route id="Init">
        <from uri="file:{{file.path}}?move=.done&amp;moveFailed=.error&amp;readLock=changed&amp;readLockCheckInterval=1500&amp;charset=UTF-8"/>
        <transacted/>

        <split streaming="true" stopOnException="true" shareUnitOfWork="true" parallelProcessing="false">
            <tokenize token="\r\n"/>

            <choice>
                <when>
                    <simple>${body.substring(0,4)} == 4000</simple>

                    [...]

                    <to uri="file:{{file.path}}/tmp?fileName=${date:now:yyyyMMddss}.txt&amp;fileExist=append&amp;charset=UTF-8"/>
                </when>

                <when>
                    <simple>${body.substring(0,4)} == 4002</simple>

                    [...]

                    <to uri="file:{{file.path}}/tmp?fileName=${date:now:yyyyMMddss}.txt&amp;fileExist=append&amp;charset=UTF-8"/>
                </when>
            </choice>

        </split>

    </route>

Second route, which consumes the file produced by the first route:

<route id="End">
        <from uri="file:{{file.path}}/tmp?delete=true&amp;moveFailed=.error&amp;readLock=changed&amp;readLockCheckInterval=1500&amp;charset=UTF-8"/>
        <transacted/>

        <split streaming="true" stopOnException="true" shareUnitOfWork="true" parallelProcessing="false">
            <tokenize token="\r\n4000"/>

            [...]

            <to uri="[...]"/>

        </split>

    </route>

I am trying to make sure file produced by route Init won't be consumed by route End until the Init has finished processing the first file.

I guessed using a temp file extension, and then using an exlude on the second route, but it doesn't work with fileExists.

Any ideas?

Thanks!

Upvotes: 1

Views: 1184

Answers (2)

Erik Karlstrand
Erik Karlstrand

Reputation: 1537

You cannot use readLock=changed with the file component as it's only available for FTP/SFTP from Camel 2.8 onwards.

changed is using file length/modification timestamp to detect whether the file is currently being copied or not. Will at least use 1 sec. to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency. This option is only avail for the FTP component from Camel 2.8 onward. Note: from Camel 2.10.1 onward the FTP option fastExistsCheck can be enabled to speedup this readLock strategy, if the FTP server support the LIST operation with a full file name (some servers may not).

Try one of the other mechanisms such as markerFile, fileLock, or rename

Upvotes: 0

hk6279
hk6279

Reputation: 1879

Use done file

You need a mechanism to make sure the second route only consume file that have been completely processed by first route.

A simple method is to let first route emit a done file as a signal to tell second route that the file has been processed completed and is ready to pickup.

To use done file, you could add doneFileName parameter in first route when process completed and also add in the second route using same filename pattern.

For more details, please read Section "Using 'done' Files" of Camel File Component

Upvotes: 2

Related Questions