CODEWITHSUNDEEP
javaamazon-s3
Chak
Chak

Reputation: 75

How to transfer a file between Amazon S3 buckets programmatically using java?

I'm new in Amazon s3 web service and need to develop a command line application to transfer a file between Amazon S3 buckets. The content of the input file must be converted to the target format and then copied to the destination folder. Target format can be XML or Json and file content respects a given data model.

I have intermediate experience with Java and just created an account which is still pending and hence, trying to develop a workflow to solve the problem.

Upvotes: 0

Views: 2713

Answers (1)

Ravindra Ranwala
Ravindra Ranwala

Reputation: 21124

Well, it's not that hard. I have done it to a customer few months back, and you may find the code below. To read a file from AmazonS3 bucket go through this Amazon documentation [1]. To write a file into Amazon s3 bucket read this documentation [2].

Other than that you may need to add all the access tokens into your local Operating system. You may get some help from an Admin person to do that. Getting the correct credentials is the only tricky part as I remember.

Amazon has a nice little documentation and I recommend you to go through that too.

package org.saig.watermark.demo;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.net.URL;

import org.apache.commons.io.IOUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import com.amazonaws.AmazonClientException;
import com.amazonaws.HttpMethod;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.S3Object;

public class AmazonS3Util {
    private static AWSCredentials credentials = null;
    private static final String fileSeparator = "/";
    private static final Log log = LogFactory.getLog(AmazonS3Util.class);
    static {
        /*
         * The ProfileCredentialsProvider will return your [default]
         * credential profile by reading from the credentials file located at
         * (~/.aws/credentials).
         */
        try {
            credentials = new ProfileCredentialsProvider().getCredentials();
        } catch (Exception e) {
            throw new AmazonClientException(
                                            "Cannot load the credentials from the credential profiles file. "
                                                    + "Please make sure that your credentials file is at the correct "
                                                    + "location (~/.aws/credentials), and is in valid format.",
                                            e);
        }
    }

    public static void readFileFromS3cketBucket(String bucketName, String key, String dirPath,
                                                String fileName) {
        FilterInputStream inputStream = null;
        FileOutputStream outputStream = null;
        try {
            // Remove the file if it already exists.
            if (new File(dirPath + WatermarkConstants.fileSeparator + fileName).exists()) {
                FileUtil.delete(new File(dirPath + WatermarkConstants.fileSeparator + fileName));
            }

            AmazonS3 s3 = new AmazonS3Client(credentials);
            Region usEast1 = Region.getRegion(Regions.US_EAST_1);
            s3.setRegion(usEast1);
            log.info("Downloading an object from the S3 bucket.");
            S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
            log.info("Content-Type: " + object.getObjectMetadata().getContentType());
            inputStream = object.getObjectContent();

            File dirForOrder = new File(dirPath);
            if (!dirForOrder.exists()) {
                dirForOrder.mkdir();
            }

            outputStream = new FileOutputStream(new File(dirPath + fileSeparator + fileName));
            IOUtils.copy(inputStream, outputStream);
            inputStream.close();
            outputStream.close();
        } catch (FileNotFoundException e) {
            log.error(e);
        } catch (IOException e) {
            log.error(e);
        }
    }

    public static void uploadFileToS3Bucket(String bucketName, String key, String dirPath,
                                            String fileName) {
        AmazonS3 s3 = new AmazonS3Client(credentials);
        Region usEast1 = Region.getRegion(Regions.US_EAST_1);
        s3.setRegion(usEast1);
        s3.putObject(new PutObjectRequest(bucketName, key, new File(dirPath + fileSeparator +
                                                                    fileName)));
        try {
            FileUtil.delete(new File(dirPath));
        } catch (IOException e) {
            log.error(e);
        }

    }

    public static void main(String[] args) {
        readFileFromS3cketBucket("bucketName",
                                 "s3Key",
                                 "localFileSystemPath",
                                 "destinationFileName.pdf");
    }
}

Hope this helps. Happy Coding !

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpJava.html

Upvotes: 3

Related Questions