CODEWITHSUNDEEP
couchdbapache-spark-sqlcloudantdata-science-experience
Romeo Kienzler
Romeo Kienzler

Reputation: 3539

How to connect to Cloudand/CouchDB using SparkSQL in DataScience Experience?

formerly CouchDB was supported via the cloudant connector:

https://github.com/cloudant-labs/spark-cloudant

But this project states that it is no longer active and that it moved to Apache Bahir:

http://bahir.apache.org/docs/spark/2.1.1/spark-sql-cloudant/

So I've installed the JAR in a Scala notebook using the following command:

%AddJar http://central.maven.org/maven2/org/apache/bahir/spark-sql-cloudant_2.11/2.1.1/spark-sql-cloudant_2.11-2.1.1.jar

Then, from a python notebook, after restarting the kernel, I use the following code to test:

spark = SparkSession\
    .builder\
    .appName("Cloudant Spark SQL Example in Python using dataframes")\
    .config("cloudant.host","0495289b-1beb-4e6d-888e-315f36925447-bluemix.cloudant.com")\
    .config("cloudant.username", "0495289b-1beb-4e6d-888e-315f36925447-bluemix")\
    .config("cloudant.password","xxx")\
    .config("jsonstore.rdd.partitions", 8)\
    .getOrCreate()

# ***1. Loading dataframe from Cloudant db
df = spark.read.load("openspace", "org.apache.bahir.cloudant")
df.cache()
df.printSchema()
df.show()

But I get:

java.lang.ClassNotFoundException: org.apache.bahir.cloudant.DefaultSource

(gist of full log)

Upvotes: 2

Views: 1415

Answers (2)

Romeo Kienzler
Romeo Kienzler

Reputation: 3539

There is one workaround, it should run in all sorts of jupyther notebook environments and is not exclusive to IBM DataScience Experience:

!pip install --upgrade pixiedust

import pixiedust

pixiedust.installPackage("cloudant-labs:spark-cloudant:2.0.0-s_2.11")

This is of course a workaround, will post the official answer once awailable

EDIT:

Don't forget the restart the jupyter kernel afterwards

EDIT 24.12.18: Created a yt video on this without workaround, see comments...will update this post as well at a later stage...

Upvotes: 3

elaver
elaver

Reputation: 151

Another workaround below. It has been tested and works in DSX Python notebooks:

import pixiedust

# Use play-json version 2.5.9. Latest version is not supported at this time.
pixiedust.installPackage("com.typesafe.play:play-json_2.11:2.5.9")
# Get the latest sql-cloudant library
pixiedust.installPackage("org.apache.bahir:spark-sql-cloudant_2.11:0")

spark = SparkSession\
  .builder\
  .appName("Cloudant Spark SQL Example in Python using dataframes")\
  .config("cloudant.host", host)\
  .config("cloudant.username", username)\
  .config("cloudant.password", password)\
  .getOrCreate()

df = spark.read.load(format="org.apache.bahir.cloudant", database="MY-DB")

Upvotes: 1

Related Questions