CODEWITHSUNDEEP
apache-camel
A5300
A5300

Reputation: 419

Is poolEnrich loading the entire file content in memory?

I have a simple route like:

from("direct:fooBar")
 .pollEnrich()
 .simple("file:in/?fileName=abc.txt")
 .process(exchange -> {
          // 1
          exchange.getIn().getBody(String.class);
 })
 .split(body().tokenize("\n")).streaming().stopOnException()
 .process(exchange -> {
          // 2
          exchange.getIn().getBody(String.class);
  })

abc.txt it's a really huge file, so I'm using split and streaming to reduce memory overhead. But I'm able to grab file content at 1, so I guess that the entire content is actually loaded into memory when using poolEnrich.

Is my assumption correct ?

Upvotes: 2

Views: 757

Answers (1)

Claus Ibsen
Claus Ibsen

Reputation: 55545

No pollEnrich does NOT load the entire file into memory. The file endpoint just gives you a file handle to the file, eg a java.io.File.

Its your code in the processor that loads the file into memory, when you tell Camel to convert the message body as a String. Then Camel will load the java.io.File into memory and give you its content as a String.

So instead of a String you can use InputStream if you want to read the body in a streaming fashion.

The splitter with the tokenizer will use java.util.Scanner to read chunks of the file and process that chunk by chunk. So that would work with the file endpoint and read it via a FileInputStream.

So remove your own code where you get the message body as a String and it ought to not load the entire file into memory.

Upvotes: 8

Related Questions