Saad Hussain
Reputation: 29
Running custom spark build on Dataproc?
Is it possible to compile and build custom Apache Spark on Google Cloud Dataproc? Lets say we want to tweak Apace Spark and then want to build custom Spark on dataproc.
Upvotes: 0
Views: 245
Answers (1)
O. Gindele
Reputation: 376
This should be possible. Dataproc spins up normal compute engine VMs for you and sets up Hadoop and YARN.
You can login to any machine, install your custom spark build and specify the YARN and Hadoop installation (of Dataproc). However, I doubt you will be able to use the GCP console or the dataproc command line interface to submit and monitor jobs with your own installation without further modification.
Upvotes: 1
Related Questions
- How to run spark 3.2.0 on google dataproc?
- Dataproc; Spark job fails on Dataproc Spark cluster, but runs locally
- Spark 2.1 on Dataproc Image
- Dataproc ignoring Spark configuration
- GCP Dataproc: Directly working with Spark over Yarn Cluster
- How can I install a custom version of Apache Spark on Cloud Dataproc
- Spark on Dataproc fails with java.io.FileNotFoundException:
- Google Cloud Dataproc migration to Spark 1.6.0
- SparkR on Dataproc (Spark 1.5.x) does not work
- Google Cloud Dataproc configuration issues