CODEWITHSUNDEEP
pythonapache-sparkpysparkhiveapache-spark-sql
Rahul Patidar
Rahul Patidar

Reputation: 209

Vertex failure While executing Pyspark Job

I am Executing One pyspark Job using Spark Submit Command.Earlier it was working file I have executed same job more than 10 Times.Its just a simple data load command from csv file to hive table and contains only 500 records.When I'm executing this same command now it showing vertex Failure Issue.

I'm Using below Spark submit Command.

    spark-submit --num-executors 3 --executor-cores 3 --executor-memory 20g  
    --jars /usr/hdp/3.1.0.0-78/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar   
    --py-files /usr/hdp/current/hive_warehouse_connector/pyspark_hwc-1.0.0.3.1.0.0-78.zip main.py 
    /user/hive/source

    /user/hive/sourc>Source File Location

I m Getting Below Error Message.

    Error while processing statement: FAILED: Execution Error, 
    return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
    vertexId=vertex_1599711935259_0207_17_00, diagnostics=[Task failed, taskId=task_1599711935259_0207_17_00_000000, 
     diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : 
    Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0,
  Vertex vertex_1500011935259_0207_17_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, 
    vertexName=Reducer 2, vertexId=vertex_1500011935259_0207_17_00, 
    diagnostics=[Vertex received Kill while in RUNNING state., 
    Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, 
   Vertex vertex_1500011935259_0207_17_00 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
   DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

can someone Please suggest me how should i resolve this error.?

Upvotes: 1

Views: 1385

Answers (1)

Rahul Patidar
Rahul Patidar

Reputation: 209

This issue got resolved I was getting this is due to mismatch calculation of following parameter.

    hive.tez.container.size
    tez.runtime.io.sort.mb=0.4*hive.tez.container.size
    tez.runtime.unordered.output.buffer.size-mb=0.1*hive.tez.container.size

Upvotes: 2

Related Questions