CODEWITHSUNDEEP
performanceapache-spark
vy32
vy32

Reputation: 29655

What does TID mean in the spark log?

Here is an example from one of my spark logs:

21/03/31 19:51:33 INFO TaskSetManager: Starting task 362467.0 in stage 1.0 (TID 362470, ip-10-252-47-180, executor 223, partition 362467, PROCESS_LOCAL, 7453 bytes)
21/03/31 19:51:33 INFO TaskSetManager: Starting task 362468.0 in stage 1.0 (TID 362471, ip-10-252-46-204, executor 127, partition 362468, PROCESS_LOCAL, 7453 bytes)
21/03/31 19:51:33 INFO TaskSetManager: Finished task 361874.0 in stage 1.0 (TID 361877) in 13503 ms on ip-10-252-46-204 (executor 127) (361652/1372624)
21/03/31 19:51:33 INFO TaskSetManager: Starting task 362469.0 in stage 1.0 (TID 362472, ip-10-252-46-14, executor 282, partition 362469, PROCESS_LOCAL, 7453 bytes)
21/03/31 19:51:33 INFO TaskSetManager: Finished task 361873.0 in stage 1.0 (TID 361876) in 13504 ms on ip-10-252-47-180 (executor 223) (361653/1372624)
21/03/31 19:51:33 INFO TaskSetManager: Finished task 361872.0 in stage 1.0 (TID 361875) in 13505 ms on ip-10-252-46-14 (executor 282) (361654/1372624)
21/03/31 19:51:33 INFO TaskSetManager: Starting task 362470.0 in stage 1.0 (TID 362473, ip-10-252-44-7, executor 421, partition 362470, PROCESS_LOCAL, 7453 bytes)

Also:

Also, I saw this wacky message:

21/03/31 19:06:23 INFO ExecutorAllocationManager: Requesting 161268 new executors because tasks are backlogged (new desired total will be 685555)

How can it request 161,268 executors? I'm running on 12 machines with 96 cores, using the default of 1 core/executor. SO I've only got 1152 cores to play around with.

Upvotes: 1

Views: 817

Answers (1)

coding happy
coding happy

Reputation: 11

about spark ,I think it should be task ID

Upvotes: 1

Related Questions