Need to run bash script in aws datapipeline

Question

i need to call the ruby file using the bash script in aws data pipeline

i have tried using shell command activity with command args

json file

> {
>       "objects": [
>         {
>           "terminateAfter": "1 Hours",
>           "id": "ResourceId5",
>           "schedule": {
>             "ref": "ScheduleId4"
>           },
>           "name": "Resource1",
>           "logUri": "s3://pipeline_test/output1/",
>           "type": "Ec2Resource"
>         },
>         {
>           "id": "ActivityId1",
>           "schedule": {
>             "ref": "ScheduleId4"
>           },
>           "name": "Shell",
>           "command": "bash -lc 'cd ~/pipeline_test/inputs/ && ruby sample.rb'", # bash command script path for ruby file
>           "runsOn": {
>             "ref": "ResourceId5"
>           },
>           "type": "ShellCommandActivity",
>           "output": {
>             "ref": "DataNodeId3"
>           }
>         },
>         {
>           "id": "DataNodeId3",
>           "schedule": {
>             "ref": "ScheduleId4"
>           },
>           "directoryPath": "s3://pipeline_test/output/",
>           "name": "Output",
>           "type": "S3DataNode"
>         },
>         {
>           "id": "Default",
>           "scheduleType": "timeseries",
>           "name": "Default",
>           "role": "DataPipelineDefaultRole",
>           "resourceRole": "DataPipelineDefaultResourceRole"
>         },
>         {
>           "id": "ScheduleId4",
>           "startDateTime": "2013-08-01T00:00:00",
>           "name": "schedule",
>           "type": "Schedule",
>           "period": "20 Minutes",
>           "endDateTime": "2013-08-03T00:00:00"
>         }
>       ]
>     }

sample.rb

f = File.open('text.txt', 'a+')
old_out = $stdout
$stdout = f
puts "Start time #{Time.now}"
puts "Welcome"
puts "End time #{Time.now}"
f.close

i dont know how to give the s3 path ("command": "bash -lc 'cd ~/pipeline_test(bucket_name)/inputs/ && ruby sample.rb'", )

i am getting the script exit status 1

Help me out to solve it.

Answer 1

One way to achieve this would be to have a wrapper shell script, something like below which invokes "sample.rb".

$INPUT1_STAGING_DIR/sample.rb >> $OUTPUT1_STAGING_DIR/output.txt

Now instead of specifying "command", you could specify "Script Uri" pointing to the shell script in S3.

You should also enable "Stage = true", and make the Input datanode point to s3 folder containing sample.rb script.

More details on Staging can be found here

You will need to modify your sample.rb and have appropriate path like "$INPUT1_STAGING_DIR/text.txt" instead of giving "text.txt".

Hope this helps.