How to read all lines in ItemReader and return lines and file address to ItemProcessor?

Question

I defined a job flow in my batch Spring project and defined ItemReader, ItemProcessor, ItemWriter, etc.

My ItemReader as below code :

@Component
@StepScope
public class MyFileReader extends FlatFileItemReader<FileInfo> {
    private String fileName;

    public MyFileReader () {
    }

    @Value("#{jobParameters[fileName]}")
    public void setFileName(final String fileName) {
        this.fileName = fileName;
    }

    @Override
    public void afterPropertiesSet() throws Exception {
        Resource resource = new FileSystemResource(fileName);
        setResource(resource);
        setEncoding("UTF-8");
        super.afterPropertiesSet();
    }
}

and my file input format is:

111111,11111,111,111  
222222,22222,222,222  

I want to read all lines of file and return lines and file address to ItemProcessor, but FlatFileItemReader read line by line. How do I do it correctly? Is overriding doRead method and handle problem manually correct?

Answer 1

If I'm understanding the question, you want to read in all lines from a file, store that data in an object and then pass said object to the processor. One approach would be to read all lines from the file before the job starts using a Job Listener. As illustrated below, you could read all lines in, populate a Java object that represents the content of a single row, collect all of those objects (so if there were two rows you'd populate 2 beans), and then pass them to the processor one at a time (or potentially at the same time, if you wish). It would look something like this:

1. First you would create a listener.

   public class MyJobListenerImpl implements JobExecutionListener {
       private MyFileReader reader;
   
   @Override
   public void beforeJob(JobExecution jobExecution) {
       reader.init();
   }
   
   @Override
   public void afterJob(JobExecution jobExecution) {
       // noop
   }
   
   // Injected
   public void setReader(MyFileReader reader) {
       this.reader = reader;
   }
   

2. Next add an init method to your custom reader.

   public void init() {
       if(Files.exists(inputFileLocation)) {
           List<String> inputData = null;
           try {
               inputData = Files.readAllLines(inputFileLocation, StandardCharsets.UTF_8);
           } catch(IOException e) {
               System.out.println("issue reading input file {}. Error message: {}", inputFileLocation, e);
               throw new IllegalStateException("could not read the input file.");
           }
   
           try {
               for(String fileItem : inputData) {
   
                   YourFileDataBean fileData = new YourFileDataBean();
   
                   yourFileDataBean.setField1(fileItem.split(",")[0].trim());
                   yourFileDataBean.setFiled2(fileItem.split(",")[1].trim());
                   yourFileDataBean.setField3(fileItem.split(",")[2].trim());
                   yourFileDataBean.setField4(fileItem.split(",")[3].trim());
                   myDeque.add(yourFileDataBean);  // really up to you how you want to store your bean but you could add a Deque instance variable and store it there.
   
           }
       } catch(ArrayIndexOutOfBoundsException e) {
           LOGGER.warn("ArrayIndexOutOfBoundsException due to data in input file.");
           throw new IllegalStateException("Failure caused by init() method. Error reading in input file.");
       }
   } else {
       LOGGER.warn("Input file {} does not exist.", inputFileLocation);
       throw new IllegalStateException("Input file does not exist at file location " + inputFileLocation);
   }
   }
   

3. Make your read() (or MyFileReader()) method in your custom reader return the object populated by all the file lines read in. In this example I am implementing ItemReader rather than extending it as you have done, but you get the idea. And if you intend to return a single Java object that represents the entire file then there would be no need to store the object in a Deque or List.

   @Override
   public MyFileReader read() throws NonTransientResourceException {
   
       return myDeque.poll();
   }
   

   Hope this helps.

As for returning the file address to the ItemProcessor. You could make this a field in YourFileDataBean and store inputFileLocation there, or save it to the execution context and access it that way. If you inject this file path into your reader, you could do the same in your processor assuming your reader plays no role in determining the file path (aka, it's predetermined).