CODEWITHSUNDEEP
multithreadingapache-sparkhadoop-yarn
Nicola Ferraro
Nicola Ferraro

Reputation: 4189

Is multithreading allowed on Spark/YARN?

Since YARN is responsible of managing the resources of Hadoop applications, like cores and processes, is a YARN application allowed to create new threads ?

I refer mainly to Spark applications running on YARN resource manager. I know that if you create a new thread no errors are thrown but, is it safe ?

Upvotes: 8

Views: 2667

Answers (1)

Nicola Ferraro
Nicola Ferraro

Reputation: 4189

I reply to myself. YARN controls resources like memory and cores, not threads. Cores are controlled by changing OS specific parameters when creating containers (process affinity). Then, the JVM is running using a specified number of cores and you can create as many threads as you want.

Upvotes: 8

Related Questions