CODEWITHSUNDEEP
apache-sparkpysparkazure-synapse
Michael
Michael

Reputation: 1

Allocation of Executors in Callee/Caller Spark Notebooks

I am working in Azure Synapse analytics and have a wrapper notebook calling two other notebooks, so am running something like:

mssparkutils.notebook.run('/notebook_1_path', 3600)

mssparkutils.notebook.run('/notebook_2_path', 3600)

How does spark allocate executors to called notebooks? Does it base it off of the configuration settings of the caller or callee notebook?

As a natural follow-up, if my caller notebook is doing nothing else but calling these two notebooks, what is the ideal configuration setup for these three notebooks combined? Is it to configure all notebooks to use the maximum number of executors?

Upvotes: 0

Views: 23

Answers (0)

Related Questions