CODEWITHSUNDEEP
javaapache-cameltransformation
Naresh Chaurasia
Naresh Chaurasia

Reputation: 469

Best way to convert the content of a big file?

I am learning camel and while trying out examples for file transformation (comma separated to colon separated), I came across the following example:

Note: The entire code is available at following location: https://github.com/dilipsundarraj1/TeachApacheCamel/blob/master/learncamel-transform

public class CamelModifyFileProcessorRoute extends RouteBuilder {
    public void configure() throws Exception {
        from("file:data/input?noop=true")
                .process(new CamelFileExampleProcessor())
                .log("Received Message is ${body} and Headers are ${headers}")
                .to("file:data/output?fileName=output.txt");
    }
}

CamelFileExampleProcessor

In the following code, the file is being transformed in-memory.

public class CamelFileExampleProcessor implements org.apache.camel.Processor {

    String newValue="";
    private String readLine;

    public void process(Exchange exchange) throws Exception {

        GenericFile<File> file = (GenericFile) exchange.getIn().getBody();
        String readLine=null;

        if(file!=null){

            FileReader file1 = new FileReader(file.getFile());
            BufferedReader reader = new BufferedReader(file1);

            while((readLine =reader.readLine()) !=null){
                System.out.println("Read line is : " + readLine);

                String oldValue = readLine;

                System.out.println("Old Value is : " + oldValue);

                newValue = newValue.concat(oldValue.replace(",", ":")).concat("\n");

                System.out.println("New Value is :  " + newValue);

                exchange.getIn().setBody(newValue);
            }

        }

    }
}

Question: In case the file size is very big, is this a good approach to follow or are there better alternatives?

Upvotes: 1

Views: 337

Answers (1)

Nicolas Filotto
Nicolas Filotto

Reputation: 44965

In case of a big file, your approach is not appropriate as it keeps the whole content of the file in memory.

What you should do instead is to use the Splitter EIP with \n as separator in streaming mode to split the content of your file by line with a very low memory footprint as next:

from("file:data/input?noop=true")
    // Split the content of the file by line in streaming mode
    .split(body().tokenize("\n")).streaming()
        // Replace the commas with colons in the current line
        .transform(simple("${body.replace(',',':')}"))
        // Append the new converted line to the target file
        .to("file:data/output?fileExist=Append") 
    .end()
    .log("File converted");

For better performances, you can also consider doing your conversion by chunks simply by grouping n lines together, it consumes more memory but it is faster. The best size of group will depend on your heap size and the content of your lines.

Assuming that you want to convert your file by chunk of 10 lines, the code would then be:

from("file:data/input?noop=true")
    // Split the content of the file by 10 lines in streaming mode 
    // without skipping the first line
    .split(body().tokenize("\n", 10, false)).streaming()
        // Replace the commas with colons in the current lines
        .transform(simple("${body.replace(',',':')}"))
        // Append the new converted lines to the target file
        .to("file:data/output?fileExist=Append") 
    .end()
    .log("File converted");

Upvotes: 1

Related Questions