CODEWITHSUNDEEP
javaapache-sparkkubernetesremote-accessremote-server
Kirill Linnik
Kirill Linnik

Reputation: 91

Can't connect to Spark running in Kubernetes from Java

I have installed Kuberenetes (minikube for Windows 10) and added Spark there using helm: 

.\helm.exe install --name spark-test stable/spark

Then I exposed Spark master port 7077 using

.\kubectl.exe expose deployment spark-test-master --port=7070 --name=spark-master-ext --type=NodePort

For example, my UI runs on http://<MINIKUBE_IP>:31905/ and spark master is exposed to <MINIKUBE_IP>:32473. In order to check, I do:

.\minikube-windows-amd64.exe service spark-master-ext

But when I do in Java:

SparkConf conf = new SparkConf().setMaster("spark://192.168.1.168:32473").setAppName("Data Extractor");

I've got:

18/03/19 13:57:29 WARN AppClient$ClientEndpoint: Could not connect to 192.168.1.168:32473: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:32473]
18/03/19 13:57:29 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://[email protected]:32473] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:32473]] Caused by: [Connection refused: no further information: /192.168.1.168:32473]
18/03/19 13:57:29 WARN AppClient$ClientEndpoint: Failed to connect to master 192.168.1.168:32473
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://[email protected]:32473/), Path(/user/Master)]

Any ideas, how to run Java Spark jobs on Spark running in Minikube?

Upvotes: 0

Views: 212

Answers (1)

Kirill Linnik
Kirill Linnik

Reputation: 91

It looks like Helm chart for Spark is really outdated (1.5.1), so I have installed 2.3.0 locally and it runs without any issues. Case closed, sorry :)

Upvotes: 1

Related Questions