CODEWITHSUNDEEP
springspring-boothdfsspring-batch
rupesh
rupesh

Reputation: 433

Reading file from HDFS using Spring batch

I've to write a Spring batch which will read a file from HDFS and will update the data in MySQL DB.

The source file in HDFS contains some report data, in CSV format.

Can someone point me to an example of reading a file from HDFS?

Thanks.

Upvotes: 3

Views: 1954

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31650

The FlatFileItemReader in Spring Batch works with any Spring Framework Resource implementation:

@Bean
public FlatFileItemReader<String> itemReader() {
    Resource resource; // get (or autowire) resource
    return new FlatFileItemReaderBuilder<String>()
            .resource(resource)
            // set other reader properties
            .build();
}

So if you manage to have a Resource handle pointing to a HDFS file, your are done.

Now in order to have a HDFS resource, you can:

  • Use Spring for Hadoop. Once the HDFS file system is configured, you would be able to get the resource from the application context with applicationContext.getResource("hdfs:data.csv");
  • Implement your own Resource using Hadoop APIs (like shown in the answer by Michael Simons). I see that some folks already did this here

Hope this helps.

Upvotes: 2

Related Questions