CODEWITHSUNDEEP
google-cloud-platformdask
Mojtaba
Mojtaba

Reputation: 11

Running a Dask script on a GCP cluster

I'm new to #GCP and my question may seem trivial (so sorry for that). I already developed a distributed version of my python code using Dask. Now, I'm going to check the scalability and the performance of my code on #Google_Cloud. To do that, I'm going to use "dataproc" for creating a cluster. But my issue is, I do not know how to run my python-dask script on the cluster. As far as I know, I should create a script for initialization on dataproc, I know how to set an initialization script for installing conda or other software or libraries, but writing a script in such a way that could be able to run a dask code on a cluster is a little vague to me. Can any body explain an example of that initialization script for running a sample python-dask code in a clusetr with 1 master node and 2 clients?

Upvotes: 0

Views: 992

Answers (1)

MRocklin
MRocklin

Reputation: 57251

Google's DataProc service is a managed YARN cluster. For this you probably want to use the Dask Yarn project.

Upvotes: 2

Related Questions