CODEWITHSUNDEEP
javaamazon-web-serviceshadoopamazon-emr
S.ziky
S.ziky

Reputation: 1

MapReduce input from HTable on AWS timeout

I'm having a bit of trouble figuring out how to excute a simple MapReduce job with input to be sourced from HTable using emr-5.4.0. when I ran on ERM, it failed because of time out.(emr-5.3.0 also failed)

I have done a bunch of google searching to find out how to proceed, but could'nt find anything useful.

my process：

  1. I created a EMR cluster using Hbase.The version is:

Amazon 2.7.3, Ganglia 3.7.2, HBase 1.3.0, Hive 2.1.1, Hue 3.11.0, Phoenix 4.9.0

  1. According to the sample from the manual:http://hbase.apache.org/book.html#mapreduce.example, and write my job likes:

    public class TableMapTest3 {

    // TableMapper
public static class MyMapper extends TableMapper<Text, Text> {

    protected void map(ImmutableBytesWritable key, Result inputValue, Context context)
            throws IOException, InterruptedException {
        String keyS = new String(key.get(), "UTF-8");
        String value = new String(inputValue.getValue(Bytes.toBytes("contents"), Bytes.toBytes("name")), "UTF-8");
        System.out.println("TokenizerMapper :" + value);
        context.write(new Text(keyS), new Text(value));
    }
}

public static void main(String[] args) throws Exception {
    Configuration conf = HBaseConfiguration.create();
    System.out.println("url:" + conf.get("fs.defaultFS"));
    System.out.println("hbase.zookeeper.quorum:" + conf.get("hbase.zookeeper.quorum"));
    Connection conn = ConnectionFactory.createConnection(conf);

    Admin admin = conn.getAdmin();
    String tableName = "TableMapTest";
    TableName tablename = TableName.valueOf(tableName);

    Table hTable = null;
    // check table exists
    if (admin.tableExists(tablename)) {
        System.out.println(tablename + " table existed...");
        hTable = conn.getTable(tablename);
        ResultScanner resultScanner = hTable.getScanner(new Scan());
        for (Result result : resultScanner) {
            Delete delete = new Delete(result.getRow());
            hTable.delete(delete);
        }
    } else {
        HTableDescriptor tableDesc = new HTableDescriptor(tablename);
        tableDesc.addFamily(new HColumnDescriptor("contents"));
        admin.createTable(tableDesc);
        System.out.println(tablename + " table created...");
        hTable = conn.getTable(tablename);
    }

    // insert data
    for (int i = 0; i < 20; i++) {
        Put put = new Put(Bytes.toBytes(String.valueOf(i)));
        put.addColumn(Bytes.toBytes("contents"), Bytes.toBytes("name"), Bytes.toBytes("value" + i));
        hTable.put(put);
    }
    hTable.close();

    // Hadoop
    Job job = Job.getInstance(conf, TableMapTest3.class.getSimpleName());
    job.setJarByClass(TableMapTest3.class);
    job.setOutputFormatClass(NullOutputFormat.class);

    Scan scan = new Scan();
    TableMapReduceUtil.initTableMapperJob(tableName, scan, MyMapper.class, Text.class, Text.class, job);

    System.out.println("TableMapTest result:" + job.waitForCompletion(true));
}

    }

  2. package my source to jar and upload it to the cluster. then I ssh on the master and run my job:

    hadoop jar zz-0.0.1.jar com.ziki.zz.TableMapTest3

I got the follow messages:

url:hdfs://ip-xxx.ap-northeast-1.compute.internal:8020
hbase.zookeeper.quorum:localhost
TableMapTest table created...
17/05/05 01:31:23 INFO impl.TimelineClientImpl: Timeline service address: http://ip-xxx.ap-northeast-1.compute.internal:8188/ws/v1/timeline/
17/05/05 01:31:23 INFO client.RMProxy: Connecting to ResourceManager at ip-xxx.ap-northeast-1.compute.internal/172.31.4.228:8032
17/05/05 01:31:24 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/05/05 01:31:31 INFO mapreduce.JobSubmitter: number of splits:1
17/05/05 01:31:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1493947058255_0001
17/05/05 01:31:33 INFO impl.YarnClientImpl: Submitted application application_1493947058255_0001
17/05/05 01:31:34 INFO mapreduce.Job: The url to track the job: http://ip-xxx.ap-northeast-1.compute.internal:20888/proxy/application_1493947058255_0001/
17/05/05 01:31:34 INFO mapreduce.Job: Running job: job_1493947058255_0001
17/05/05 01:31:57 INFO mapreduce.Job: Job job_1493947058255_0001 running in uber mode : false
17/05/05 01:31:57 INFO mapreduce.Job:  map 0% reduce 0%

after a well, i get the error:

17/05/05 01:42:26 INFO mapreduce.Job: Task Id : attempt_1493947058255_0001_m_000000_0, Status : FAILED
AttemptID:attempt_1493947058255_0001_m_000000_0 Timed out after 600 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/05/05 01:52:56 INFO mapreduce.Job: Task Id : attempt_1493947058255_0001_m_000000_1, Status : FAILED
AttemptID:attempt_1493947058255_0001_m_000000_1 Timed out after 600 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

and some syslogs

2017-05-05 01:31:59,664 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1493947058255_0001_m_000000 Task Transitioned from SCHEDULED to RUNNING
2017-05-05 01:32:08,168 INFO [Socket Reader #1 for port 33348] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1493947058255_0001 (auth:SIMPLE)
2017-05-05 01:32:08,227 INFO [IPC Server handler 0 on 33348] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1493947058255_0001_m_000002 asked for a task
2017-05-05 01:32:08,231 INFO [IPC Server handler 0 on 33348] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1493947058255_0001_m_000002 given task: attempt_1493947058255_0001_m_000000_0
2017-05-05 01:42:25,382 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1493947058255_0001_m_000000_0: AttemptID:attempt_1493947058255_0001_m_000000_0 Timed out after 600 secs
2017-05-05 01:42:25,389 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1493947058255_0001_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2017-05-05 01:42:25,392 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1493947058255_0001_01_000002 taskAttempt attempt_1493947058255_0001_m_000000_0
2017-05-05 01:42:25,392 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1493947058255_0001_m_000000_0
2017-05-05 01:42:25,394 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : ip-xxx.ap-northeast-1.compute.internal:8041
2017-05-05 01:42:25,457 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1493947058255_0001_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2017-05-05 01:42:25,458 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2017-05-05 01:42:25,460 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1493947058255_0001_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2017-05-05 01:42:25,495 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved ip-xxx.ap-northeast-1.compute.internal to /default-rack
2017-05-05 01:42:25,500 INFO [Thread-83] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node ip-xxx.ap-northeast-1.compute.internal
2017-05-05 01:42:25,502 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1493947058255_0001_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-05-05 01:42:25,503 INFO [Thread-83] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1493947058255_0001_m_000000_1 to list of failed maps
2017-05-05 01:42:25,557 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:3 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:1 RackLocal:0
2017-05-05 01:42:25,582 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1493947058255_0001: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:1024, vCores:1> knownNMs=2
2017-05-05 01:42:25,582 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1493947058255_0001_01_000002
2017-05-05 01:42:25,583 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1493947058255_0001_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

I just use the default settings and run a simple job.why these errors happend? if i'm missing anything, let me know! anyway, thanks for your help - appreciate it!

Upvotes: 0

Views: 289

Answers (1)

S.ziky
S.ziky

Reputation: 1

I found the answer from : here

you cant use HConfiguration (because it defaults to a localhost quorum) What you'll have to do is use the configuration that amazon sets up for you (located in /etc/hbase/conf/hbase-site.xml)

The connection code looks like this:

    Configuration conf = new Configuration();
    String hbaseSite = "/etc/hbase/conf/hbase-site.xml";
    conf.addResource(new File(hbaseSite).toURI().toURL());
    HBaseAdmin.checkHBaseAvailable(conf);

Upvotes: 0

Related Questions