CODEWITHSUNDEEP
apache-sparkcassandrajarspark-cassandra-connector
Sajeed
Sajeed

Reputation: 139

Spark job can't connect to cassandra when ran from a jar

I have spark job that writes data to Cassandra(Cassandra is on GCP). When I run this from IntelliJIDEA (my IDE) it works perfectly fine. The data is perfectly sent and written to Cassandra. However, this fails when I package my project into a fat jar and run it.

Here is an example of how I run it.

spark-submit --class com.testing.Job --master local out/artifacts/SparkJob_jar/SparkJob.jar 1 0

However, this fails for me and gives me the following errors

Caused by: java.io.IOException: Failed to open native connection to Cassandra at {X.X.X:9042} :: 'com.datastax.oss.driver.api.core.config.ProgrammaticDriverConfigLoaderBuilder com.datastax.oss.driver.api.core.config.DriverConfigLoader.programmaticBuilder()'


Caused by: java.lang.NoSuchMethodError: 'com.datastax.oss.driver.api.core.config.ProgrammaticDriverConfigLoaderBuilder com.datastax.oss.driver.api.core.config.DriverConfigLoader.programmaticBuilder()'

My artifacts file does include the spark-Cassandra files

spark-cassandra-connector-driver_2.12-3.0.0-beta.jar
spark-cassandra-connector_2.12-3.0.0-beta.jar

I'm wondering why this is happening and how I can fix it?

Upvotes: 1

Views: 457

Answers (1)

Alex Ott
Alex Ott

Reputation: 87069

The problem is that besides that 2 things, you need to have more jars - full Java driver, and its dependencies. You have following possibilities to fix that:

  1. You need to make sure that these artifact is packaged into the resulting jar (so-called "fat jar" or an "assembly") using Maven or SBT, or anything else

  2. you can can specify maven coordinates com.datastax.spark:spark-cassandra-connector_2.12:3.0.0-beta with --packages like this --packages com.datastax.spark:spark-cassandra-connector_2.12:3.0.0-beta

  3. you can download spark-cassandra-connector-assembly artifact to the node from which you're doing spark-submit, and then use that file name with --jars

See the documentation for Spark Cassandra Connector for more details.

Upvotes: 1

Related Questions