CODEWITHSUNDEEP
apache-camel
ChrisGeo
ChrisGeo

Reputation: 3907

Apache Camel - Copying a large file into a consumer folder

I have a route that expects that various files will be copied into an incoming folder. The route will proceed to move these files into a temp folder where it will do other stuff. The route is as follows: 

       <route id="incoming" >
            <from uri="file://my/path/incoming"/>
            <to uri="file://my/path/incoming/temp"/>
        </route>

The issue is that these files may be quite large. Lets say 1Gb. In order to copy this file in to the incoming folder it may take lets say 10 seconds. During these 10 seconds the Consumer polls the directory and an exception is thrown since the partial file is still being copied. What workaround could I use?

I have used readLock all strategies (primarily changed) but I get an exception:

(The process cannot access the file because it is being used by another process)

The modified uri is as follows: 

<from uri="file://my/file/path?readLockCheckInterval=3000&amp;readLock=changed"/>

Still no luck though

Upvotes: 5

Views: 9118

Answers (4)

bvyas
bvyas

Reputation: 451

Probably this is late in the game but use fileExist=Append in the route URI. Example:

<route id="incoming" >
    <from uri="file://my/path/incoming"/>
    <to uri="file://my/path/incoming/temp?fileExist=Append"/>
</route>

Upvotes: 1

zikzakjack
zikzakjack

Reputation: 1072

Something like this will work out. Just in case if its NON-Camel system is copying your large file into the InputDir, then you have to take care to create the .DONE file after the file is copied. Once the .DONE file is available the route will start processing.

    from("file://" + InputDir + "?delay=500&doneFileName=${file:name}.DONE")
    .to("file://" + OutputDir + "?fileName=${date:now:yyyyMMdd}/${file:name}&doneFileName=${file:name}.DATA.READY.DONE");

Upvotes: 1

techuser soma
techuser soma

Reputation: 4877

readLock=changed option seems appropriate in this case. There can be issues if you have a very slow producer writing files to the incoming folder.

Other option is to use the done file name. You can make the original producer create a done file after file write is completed.

its more common to have one done file per target file. This means there is a 1:1 correlation. To do this you must use dynamic placeholders in the doneFileName option. Currently Camel supports the following two dynamic tokens: file:name and file:name.noext which must be enclosed in ${ }. The consumer only supports the static part of the done file name as either prefix or suffix (not both).

from("file:bar?doneFileName=${file:name}.done");
In this example onlyfiles will be polled if there exists a done file with the name file name.done.

Upvotes: 1

Ricardo Veguilla
Ricardo Veguilla

Reputation: 3155

Check the readLock options in the File component

Used by consumer, to only poll the files if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written). Camel will wait until the file lock is granted.

This option provides the build in strategies:

markerFile Camel creates a marker file (fileName.camelLock) and then holds a lock on it.

changed is using file length/modification timestamp to detect whether the file is currently being copied or not. Will at least use 1 sec. to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency.

fileLock is for using java.nio.channels.FileLock. This approach should be avoided when accessing a remote file system via a mount/share unless that file system supports distributed file locks.

rename is for using a try to rename the file as a test if we can get exclusive read-lock.

Upvotes: 6

Related Questions