Running Spring Batch Job as soon as the File is available

Question

I want to run my Spring Batch job as soon as the File is available in a shared folder. I have made a watcher service to check the new entry in the directory, but how would I trigger my batch Job ?

I have did the following changes -

  @Component
public class ScheduleJob {

    private Logger log = LoggerFactory.getLogger(this.getClass());

    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    private UtilizationBatchConfiguration utilizationBatchConfiguration;

    @Value("${excel.to.database.job.source.file.source.path}")
    private String PROPERTY_EXCEL_SOURCE_FILE_PATH;


    //@Scheduled(cron = "${excel.to.database.job.cron}")
    public void runJob() {
        //String fileSource = PROPERTY_EXCEL_SOURCE_FILE_PATH.concat(PROPERTY_EXCEL_SOURCE_FILE_NAME)+".xlsx";
        Path path = Paths.get(PROPERTY_EXCEL_SOURCE_FILE_PATH);
        WatchKey key;
        WatchService watchService = null;
        try {
            watchService = FileSystems.getDefault().newWatchService();
            path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);

            while ((key = watchService.take()) != null) {
                for (WatchEvent<?> event : key.pollEvents()) {

                    System.out.println(
                            "Event kind:" + event.kind()
                                    + ". File affected: " + event.context() + ".");
                    if(event.kind().equals("ENTRY_CREATE")) {
                        impoerJob();
                    }
                }
                key.reset();
            }
        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
        }
    }

    private void impoerJob() {

        // make unique JobParameters so now instance of job can be started
        Map<String, JobParameter> confMap = new HashMap<String, JobParameter>();
        confMap.put("time", new JobParameter(System.currentTimeMillis()));
        JobParameters jobParameters = new JobParameters(confMap);
        try {
            JobExecution ex = jobLauncher.run(utilizationBatchConfiguration.importExcelJob(), jobParameters);
            log.info(String.format("Execution status-----> %s, Execution Start Time ------> %s, Execution End Time %s", ex.getStatus(), ex.getStartTime(), ex.getEndTime()));
        } catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException
                | JobParametersInvalidException e) {
            //
            e.printStackTrace();
        }
    }
}

Here if I use scheduler instead of WatchService my batch job runs successfully. The problem here is that , I don't know when the file will get copy to the shared location. So have to use WatchService instead of Scheduler. And my import job is -

@Component
public class UtilizationBatchConfiguration {
---- removed for brevity

@Bean
    public Job importExcelJob() {
        return jobBuilderFactory.get("importExcelJob")
            .incrementer(new RunIdIncrementer())
            .preventRestart()
            .listener(listener)
            .flow(step1())
            .end()
            .build();
    }
}

This is almost similar to Spring Batch service guide in Spring.io

Right now If I try to run the code , I get below response -

Started Application in 10.506 seconds (JVM running for 11.676) 2018-07-23 16:55:53.646 INFO 18200 --- [ Thread-2] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@7b9a4292: startup date [Mon Jul 23 16:55:44 IST 2018]; root of context hierarchy 2018-07-23 16:55:53.649 INFO 18200 --- [ Thread-2] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown 2018-07-23 16:55:53.650 INFO 18200 --- [
Thread-2] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans 2018-07-23 16:55:53.654 INFO 18200 --- [
Thread-2] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default' 2018-07-23 16:55:53.655 INFO 18200 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated... 2018-07-23 16:55:53.683 INFO 18200 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.

Also how would I make sure that I am reading a new file everytime , instead of reading the old one again ?

Answer 1

I can't speak for the WatchService but usually a JVM needs at least one non-daemon thread running to prevent it from shutting down. It looks like your application doesn't have that (so it's shutting down). It's common to use Spring Integration for this use case and we have this in our documentation here: https://docs.spring.io/spring-batch/trunk/reference/html/springBatchIntegration.html#launching-batch-jobs-through-messages

Answer 2

I’m not completely sure why you would need Batch Job capability here as it seems that a Watcher Service would suffice.

If you have write permission to the location, you can either, tag the processed files (maybe add a prefix) and change your program to ignore file names with that prefix or you can create a processed directory and move processed files to that directory after processing or you can use use timestamp by keeping track of the last successful file process run and look for files newer than that timestamp.

Answer 3

You can use it this way, your only problem is, that your Spring-Context closes right after it is started because there is "nothing to do". Just take a look at this solution how to prevent your Spring-Context from closing.