CODEWITHSUNDEEP
bashamazon-web-servicesaws-cloudformationamazon-emr
Prabhakar Reddy
Prabhakar Reddy

Reputation: 5124

How to add EMR step with bash command to cloudformation

I am trying to add below step to my EMR cluster via Cloudformation but it fails saying file not found error. I tried escaping the double quotes but no use.

 SetupHiveStep:
        Type: AWS::EMR::Step
        Properties:
            Name: SetupHive
            ActionOnFailure: CANCEL_AND_WAIT
            JobFlowId: !Ref Cluster
            HadoopJarStep:
                Jar: "command-runner.jar"
                Args:
                    - !Sub bash -c "aws s3 cp s3://test-emr/scripts/init.sh /home/hadoop/init.sh;sudo chmod +x /home/hadoop/init.sh;sh /home/hadoop/init.sh;"

Here is the actual step configuration from EMR :

JAR location :command-runner.jar
Main class :None
Arguments :bash -c "aws s3 cp s3://test-emr/scripts/init.sh /home/hadoop/init.sh;sudo chmod +x /home/hadoop/init.sh;sh /home/hadoop/init.sh;"
Action on failure:Terminate cluster

Below is the error from EMR failed step:

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "bash -c "aws s3 cp s3://test-emr/scripts/init.sh /home/hadoop/init.sh;sudo chmod +x /home/hadoop/init.sh;sh /home/hadoop/init.sh;"" (in directory "."): error=2, No such file or directory
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:139)
    at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.io.IOException: Cannot run program "bash -c "aws s3 cp s3://preddy-test-emr/scripts/init.sh /home/hadoop/init.sh;sudo chmod +x /home/hadoop/init.sh;sh /home/hadoop/init.sh;"" (in directory "."): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:92)
    ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 8 more

Upvotes: 0

Views: 480

Answers (1)

Prabhakar Reddy
Prabhakar Reddy

Reputation: 5124

I was able to make this work by modifying the command as shown below. Had to add && along with bash -c

SetupHiveStep:
        Type: AWS::EMR::Step
        Properties:
            Name: SetupHive
            ActionOnFailure: CANCEL_AND_WAIT
            JobFlowId: !Ref Cluster
            HadoopJarStep:
                Jar: "command-runner.jar"
                Args:
                    - "bash"
                    - "-c" 
                    - !Sub "aws s3 cp s3://${S3ConfPrefix}/scripts/init.sh /home/hadoop/init.sh && sudo chmod +x /home/hadoop/init.sh && sh /home/hadoop/init.sh"
                    - !Ref S3ConfPrefix

Upvotes: 1

Related Questions