CODEWITHSUNDEEP
apache-camel
Chris
Chris

Reputation: 492

Camel File Consumer - leave file after processing but accept files with same name

So this is the situation:

I have a workflow, that waits for files in a folder, processes them and then sends them to another system.
For different reasons we use an ActiveMQ Broker between "sub-processes" in the workflow, where each route alters the message in some way before it is sent in the last step. Each "sub-processes" only reads and writes to/from the ActiveMQ, except the first and last route.

It is also part of the workflow, that there is a route after sending the message, that takes care of initial file, moving or deleting it. Only this route knows that to do with the file.

This means, that the file has to stay in the folder after the consumer-route has finished, because the meta-data is just written into the ActiveMQ, but the actual workflow is not done yet.

It got this to work using the noop=true parameter on the file consumer.
The problem with this is, that after the "After Sending Route" deletes (or moves) the file, the file consumer will not react to new files with the same name again until I restart the route.
It is clear, that this is the expected and correct behavior, because its the point of the noop parameter to ignore a file that was consumed before, but this doesn´t help me.

The question is now how I get the file consumer to only process a file once as long as it is present in the folder, but "forget" about it as soon as some other process (in this case a different route) removes the file.

As an alternative I could let the file component move the file into a temp folder, from where it gets processed later and leave the cosuming folder empty, but this introduces new problems, that I'd like to avoid (e.g. moving a file with the same name into the folder, as long as the first one is not yet processed completely)

I'd love to hear some ideas on how to handle that case.

Greets Chris

Upvotes: 1

Views: 986

Answers (1)

Screwtape
Screwtape

Reputation: 1367

You need to tell Camel to not only use the filename for idempotency checking.

In a similar situation, where I wanted to pick up changes to a file that was otherwise no-oped, I have the option

idempotentKey=${file:name}-${file:modified}

in my url, which ensures if you change the file, or a new file is created, it treats that as a different file and processes it.

Do be careful to check how many files you might be processing because the idempotent buffer is limited by default (to 1000 records I think), so if you were potentially processing more than 1000 files at a time, it might "forget" it's already processed file 1, when file 1001 arrives, and try and reprocess file 1 again.

Upvotes: 4

Related Questions