CODEWITHSUNDEEP
mongodbapache-sparkpyspark
mox601
mox601

Reputation: 432

How to change a mongo-spark connection configuration from a databricks python notebook

I succeeded at connecting to mongodb from spark, using the mongo-spark connector from a databricks notebook in python.

Right now I am configuring the mongodb uri in an environment variable, but it is not flexible, since I want to change the connection parameter right in my notebook.

I read in the connector documentation that it is possible to override any values set in the SparkConf.

How can I override the values from python?

Upvotes: 3

Views: 1020

Answers (1)

Ross
Ross

Reputation: 18101

You don't need to set anything in the SparkConf beforehand*. You can pass any configuration options to the DataFrame Reader or Writer eg:

df = sqlContext.read \ 
    .option("uri", "mongodb://example.com/db.coll) \    
    .format("com.mongodb.spark.sql.DefaultSource") \
    .load()

* This was added in 0.2

Upvotes: 2

Related Questions