CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Bostonian
Bostonian

Reputation: 687

what Spark SQL is doing when below message shown

I am new to Spark SQL and I try to run a query and notice that Spark SQL get stock for a very long time.

scala> spark.time(query.collect)
[Stage 169:===========>   (44 + 4) / 58][Stage 170:>               (0 + 0) / 58]

Does anyone know the expression like 169,(44 + 4) / 58 or (0 + 0) / 58) indicate?

Upvotes: 0

Views: 56

Answers (1)

Gaurang Shah
Gaurang Shah

Reputation: 12910

it indicates following thing 

[Stage stageNo: (numCompletedTasks + numActiveTasks) / totalNumOfTasksInThisStage]

Each Spark Job, is divided into multiple stages, and each stage is again divided into multiple tasks.

In your case. It says, 

Stage: 169 
Total Tasks: 58
Completed tasks: 44 
Active Tasks: 4 
Remaining Tasks: 10 (totalNumOfTasksInThisStage - (numCompletedTasks + numActiveTasks))

Upvotes: 3

Related Questions