CODEWITHSUNDEEP
javafilescalaapache-camelakka
MadSeb
MadSeb

Reputation: 8244

Apache Camel File Component - how to scan for new files only?

I set up a simple Apache Camel Route in Scala like this :

import org.apache.camel.builder.RouteBuilder
import akka.actor.ActorRef
import akka.camel._

    class SimpleRouteBuilder(val responder: ActorRef) extends RouteBuilder  {
      def configure  {
        from("file:C:\\datafiles\\input?noop=true&readLock=changed").to(responder)
      }
    }

My question is how do I make this only scan for new files ? Right now , it sends one message per every file in the directory. I want to have it in such a way that it only sends messages for files created after my camel application was started.

Cheers !

UPDATE :

OK I found a solution ... Implement a time stamp actor whose only purpose if to check if the ( file creation time > application start time ) and forward the file message to the round robin router.

The route is now :

 from(s"file:$directoryName?noop=true&readLock=changed&delay=$delay&recursive=true&include=$include").
  convertBodyTo(classOf[java.io.File]).to(timestampActor)

My code for the time stamp actor is approximately :

object TimestampActor {
  def apply(timeMillis0: Long): Props = Props(new TimestampActor(timeMillis0))
}

// timeMillis0 is the time when the application has started
class TimestampActor(timeMillis0: Long ) extends Actor {

  val messageRouterActor: ActorRef = context.system.actorFor(s"akka://${Constants.ActorSystemName}/user/${Constants.MessageRouterActorName}")
  val logger: Logger = LoggerFactory.getLogger(classOf[TimestampActor])


  def receive = {

      case camelMessage: CamelMessage => {
       // Need to unbox
        self ! camelMessage.body.asInstanceOf[java.io.File]
      }

      case file: File => {
        try {
          val attrs: BasicFileAttributes  = Files.readAttributes(file.toPath, classOf[BasicFileAttributes])
          if (attrs.creationTime().toMillis  >= timeMillis0) {
            // Forward the message to the Round Robin message router
            messageRouterActor ! file
          }
        }
        catch {
          case ioe:IOException => {
            logger.error(s"Failed to get creation time for file ${file.getCanonicalPath}", ioe)
            // Retry in 20 minutes
            import context.dispatcher
            context.system.scheduler.scheduleOnce(Duration.create(20, TimeUnit.MINUTES), self, file)

          }
        }
      }
  }

}

Upvotes: 1

Views: 3963

Answers (2)

tshenolo
tshenolo

Reputation: 463

If you are concerned about losing the original file I would suggest that you use multicast. This way you could send a copy of the file to a backup folder.

from("file:C:\\datafiles\\input").multicast().to("backuplocation", "responder");

By default camel will process any file that is placed in the input folder

Upvotes: 0

hveiga
hveiga

Reputation: 6925

You need to use the idempotent option of the Apache Camel File Component. It will remember the already processed files and will only use the new ones. By default it will remember 1000 entries and the key will be the absolute path of the file but you can change it to what you need. It would be something like:

from("file:C:\\datafiles\\input?noop=true&readLock=changed&idempotent=true").to(responder)

Also you can use different types of Idempotent Repositories to do more fancy stuff.

Upvotes: 4

Related Questions