CODEWITHSUNDEEP
springspring-bootspring-batchspring-batch-tasklet
Gaurav Raghav
Gaurav Raghav

Reputation: 187

Spring Batch - Read multiple files from S3

Like for reading single file in spring batch from s3, we use

@Bean
public FlatFileItemReader<Map<String, Object>> itemReader() {
    FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
    reader.setLineMapper(new JsonLineMapper());
    reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
    reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
    return reader;
}

But if i want to read all the files from some specific folder/key then is there something to MultiResourceItemReader, like below(which we use for local filesystem)

MultiResourceItemReader<UserData> reader = new MultiResourceItemReader<>();
reader.setResources(resources);

Upvotes: 1

Views: 2051

Answers (2)

Gaurav Raghav
Gaurav Raghav

Reputation: 187

Create a MultiResourceItemReader like this,

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

This reader will need a delegate and lineMapper, you can implement it like this,

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}

Upvotes: 2

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31600

No, it is up to you to create the Resource array and pass it to the MultiResourceItemReader.

Upvotes: 2

Related Questions