CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-emr
Dev
Dev

Reputation: 13773

Adding S3 sync step in EMR

After performing all the steps, I want to execute the last step to copy S3 data to another bucket.

I didn't find any supported script for running shell commands

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-commandrunner.html

s3-dist-cp is supported and I could have used it if it allows me to override target directory data.

I need something like:

{
  action_on_failure   = "CONTINUE"
  name                = "copy s3 data"
  hadoop_jar_step     = [{
     args    = ["bash" , "  aws s3 sync s3://bucket1/data s3://bucket2/data"]
     jar = "command-runner.jar"
    }]
}

Upvotes: 1

Views: 476

Answers (1)

spinatelli
spinatelli

Reputation: 46

Check out this link.

From CLI you can use this config bit for the step:

Type=CUSTOM_JAR,Name="Copy s3 data",ActionOnFailure=CONTINUE,Jar="command-runner.jar",Args=["aws","s3","sync","s3://bucket1/data","s3://bucket2/data"]

Upvotes: 2

Related Questions