Reputation: 23
fetchSize error:
TypeError: jdbc() got an unexpected keyword argument 'fetchSize'
I tried reading as
mydf = spark.read.jdbc(url, table, numPartitions=20, column=partitionColumn, lowerBound=0, upperBound=1000, fetchSize = 10000, properties=properties)
Upvotes: 1
Views: 4098
Reputation: 152
I am not sure of performance but try this below code. this will not throw error
df = (spark.read.format("jdbc").option("url", url)
.option("dbtable", "mytable")
.option("user", user) .option("password", password)
.option("numPartitions", "100").option("fetchsize","10000")
.option("partitionColumn", "id")
.option("lowerBound", "1").option("upperBound","1000000").load())
Upvotes: 4