CODEWITHSUNDEEP
apache-sparkvertica
Daksh
Daksh

Reputation: 308

Spark integration with Vertica Failing

We are using Vertica Community Edition "vertica_community_edition-11.0.1-0", and are using Spark 3.2, with local[*] master. When we are trying to save data in vertica database using following:

member.write()
                .format("com.vertica.spark.datasource.VerticaSource")
                .mode(SaveMode.Overwrite)
                .option("host", "192.168.1.25")
                .option("port", "5433")
                .option("user", "Fred")
                .option("db", "store")
                .option("password", "password")
                //.option("dbschema", "store")
                .option("table", "Test")
                //      .option("staging_fs_url", "hdfs://172.16.20.17:9820")
                .save();

We are getting following exception:

com.vertica.spark.util.error.ConnectorException: Fatal error: spark context did not exist
        at com.vertica.spark.datasource.VerticaSource.extractCatalog(VerticaDatasourceV2.scala:76)
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.getTableProviderCatalog(CatalogV2Util.scala:363)

Kindly let know how to solve the exception.

Upvotes: 2

Views: 678

Answers (2)

Andras Gabor
Andras Gabor

Reputation: 11

We had a similar case. The root cause was that SparkSession.getActiveSession() returned None, due to that spark session was registered on another thread of the JVM. We could still get to the single session we had using SparkSession.getDefaultSession() and manually register it with SparkSession.SetActiveSession(...).

Our case happened in a jupyter kernel where we were using pyspark.
The workaround code was:

sp = sc._jvm.SparkSession.getDefaultSession().get()
sc._jvm.SparkSession.setActiveSession(sp)

I can't try scala or java, I suppose it should look like this:

SparkSession.setActiveSession(SparkSession.getDefaultSession())

Upvotes: 1

geekofgeeks
geekofgeeks

Reputation: 28

vertica doesn't support spark version 3.2 with vertica 11.0 officially. Please find the below documentation link.

https://www.vertica.com/docs/11.0.x/HTML/Content/Authoring/SupportedPlatforms/SparkIntegration.htm

Please try using spark connector v2 with the supported version of spark and try running examples from github

https://github.com/vertica/spark-connector/tree/main/examples

Upvotes: 0

Related Questions