CODEWITHSUNDEEP
apache-camelrace-condition
Ivana
Ivana

Reputation: 705

Move files with Apache Camel to different directories depending on how many files are found

A process should output one and only one file, it things go wrong it could result in zero or a few files in the output directory.

I want to pick up the file(s) and move them to directory A if there was one file and to directory B if more files were found. (And send a mail to some admin).

// fileCounter and FileNameFilter use same utility class for filtering!
from(config.getFrom())
    .to("bean:bean:fileCounter?method=fileCountInHeader(*)")
    .choice()
        .when(simple("${header.NumberOfFiles} == 0"))
            .to("bean:grootboekMutatiesMailHandler?method=createMailNoFile(*)")
        .when(simple("${header.NumberOfFiles} == 1"))
            .filter().method(new FileNameFilter(), getFilterBy())
            .to(config.getA()))
            .to("bean:grootboekMutatiesMailHandler?method=createMailSuccess(*,${header.FileNamesList})")
        .when(simple("${header.NumberOfFiles} > 1"))
            .filter().method(new FileNameFilter(), getFilterBy())
            .to(config.getB())
            .to("bean:grootboekMutatiesMailHandler?method=createMailManyFiles(*,${header.FileNamesList})").endChoice().end();

If the camel route starts from file, counting the number of files will be done for each file in turn, so the last file will always be transferred to directory A.

What i really want is list the files at the start of the route, count them and then only transfer these files to the appropriate directory. If a file is added, it should be neglegted, if a file is removed an error should be thrown.

Upvotes: 1

Views: 4010

Answers (3)

Ivana
Ivana

Reputation: 705

There is no need to implement a bean to count the files at the endpoint. Getting the number of files lined up for processing is possible because the File Consumer, including the ftp Consumer, is a BatchConsumer. This means that the Batch size can be extracted from the Exchange properties.

Here is a working example of routing files depending on batch size, the example is for a file consumer but also works for ftp. 

    final String from = "file:data/input?filter=#myFilter";
    final String to1 = "file:data/output";
    final String toMany = "file:data/output2";

    from(from)
    .choice()
    .when(simple("${exchangeProperty[CamelBatchSize]} == 1"))
     .to(to1)
    .when(simple("${exchangeProperty[CamelBatchSize]} > 1"))
      .to(toMany);

Note that any filters should be set on the 'from' endpoint. Filtering along the route (ie: .filter().method(new MyFilter()) will break the batch-count-logic. Files that do not pass the filter will still be counted in CamelBatchSize! So if files aaa.txt, aa.txt and bb.txt are present at the 'from' endpoint, but only bb.txt should be processed, filtering using .filter() will only let bb.txt through but CamelBatchSize will still be 3 not 1.

Upvotes: 1

burki
burki

Reputation: 7005

I don't see how you count the number of files in the bean, but you could instead use the Camel Exchange property CamelBatchSize that returns the number of files read from the directory in one polling.

This only works if you can make sure that on every file polling you get all files from only one processing.

Another approach would be to use a timer to trigger in a given interval and then use your counter Java bean to count the files and delegate to dedicated routes for the three cases.

Something like (pseudo-code)

from(timer:countFiles?yourOptions)
    .bean(counterBean)
    .choose
        .when(0 files)
            .to(direct:noFile)
        .when(1 file)
            .to(direct:OneFile)
        .otherwise
            .to(direct:TooMuchFiles)
    .end()

In the direct routes you have various possibilities to consume the files:

  • If you are not interested in the content, but you want to just move the files, you can use a bean that does this with Java code
  • If you want to consume a single file to get its content, you can use the pollEnrich EIP
  • If pollEnrich is getting too clumsy because you want to consume a lot of files, you can write a dedicated route with a from-file (file poller) that is not started (autoStart=false). The you can use the Controlbus to start and stop this route on demand from the direct routes.

Upvotes: 1

Ryan Stuetzer
Ryan Stuetzer

Reputation: 392

You can consume the file in the direct:OneFile route by using a pollEnrich (http://camel.apache.org/content-enricher.html). If you want you could set the file name as a header value in the counter bean and use it in the pollEnrich uri.

....
from(direct:OneFile)
    .pollEnrich(<your file uri with dynamic Expression containing filename>, 0, 
                    pollEnrichAggregationStrategy)
...

Upvotes: 0

Related Questions