CODEWITHSUNDEEP
amazon-web-servicesamazon-data-pipeline
user730379
user730379

Reputation: 121

AWS data pipeline job failing but there is no error message or error code

I tried running a data pipeline job but the EmrActivity step reached a FAILED status but there is no error code or error message:

Name: @EMR cluster to perform the work_2013-09-03T16:15:00 View instance fields Description: Latest attempt count: 3, Tries left: 0 Select attempt for this instance:

Status: FAILED Error code: Error message:

any idea why? Where can I find out more info about the underlying problem?

The job is simple: fire up EMR cluster and run a pig script (where xxx is my bucket name):


    {
    "objects": [
    {
    "id":"Default",
    "failureAndRerunMode":"cascade" 
    },
    {
    "id" : "MyScheduleID",
    "type" : "Schedule",
    "period" : "1 hour",
    "startDateTime" : "2013-09-03T19:00:00",
    "endDateTime" : "2013-09-03T20:00:00"
    },
    {
    "id" : "MyEmrCluster",
    "name" : "EMR cluster to perform the work",
    "type" : "EmrCluster",
    "hadoopVersion" : "0.20",
    "masterInstanceType" : "m1.small",
    "coreInstanceType" : "m1.medium",
    "coreInstanceCount" : "2",
    "terminateAfter": "1 Hours",
    "schedule": {
    "ref": "MyScheduleID"
    },
    "logUri":"s3://xxx/amazonlogs",
    "emrLogUri":"s3://xxx/amazonlogs"
    },

    {
    "id" : "MyEmrActivity",
    "name" : "Work to perform on my data",
    "type" : "EmrActivity",
    "runsOn" : {"ref" : "MyEmrCluster"},
    "schedule": {
    "ref": "MyScheduleID"
    },
    "step": "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,s3://us-east-1.elasticmapreduce/libs/pig/pig-script,--base-path,s3://us-east-1.elasticmapreduce/libs/pig/,--install-pig,--pig-versions,latest",
    "step": "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,s3://us-east-1.elasticmapreduce/libs/pig/pig-script,--base-path,s3://us-east-1.elasticmapreduce/libs/pig/,--pig-versions,latest,--run-pig-script,--args,-f,s3://xxx/carls_minimal_script.pig"
    }

    ]
    }

Does this config look OK? I don't see anything in s3://xxx/amazonlogs

Upvotes: 1

Views: 1890

Answers (1)

Bharat Jain
Bharat Jain

Reputation: 664

Here are couple of things you could try

  1. Go to "https://console.aws.amazon.com/elasticmapreduce/home", find the corresponding cluster that got started (based on timestamp), click on "Debug", you should find logs about each step.

  2. Or start an EMR cluster from AWS Console, login into the Master node, run the Pig script to check if its working.

Upvotes: 2

Related Questions