CODEWITHSUNDEEP
postgresqlapache-sparkjdbc
Voyage Z
Voyage Z

Reputation: 23

set fetchSize in JDBC in Spark 2.x?

fetchSize error:

TypeError: jdbc() got an unexpected keyword argument 'fetchSize'

I tried reading as 

mydf = spark.read.jdbc(url, table, numPartitions=20, column=partitionColumn, lowerBound=0, upperBound=1000, fetchSize = 10000, properties=properties)

Upvotes: 1

Views: 4098

Answers (1)

user8414391
user8414391

Reputation: 152

I am not sure of performance but try this below code. this will not throw error

df = (spark.read.format("jdbc").option("url", url)
  .option("dbtable", "mytable") 
  .option("user", user) .option("password", password) 
  .option("numPartitions", "100").option("fetchsize","10000")
  .option("partitionColumn", "id")
  .option("lowerBound", "1").option("upperBound","1000000").load())

Upvotes: 4

Related Questions