fokoenecke
Reputation: 53
How to set row batch size for incrementalCollect in Apache Spark Thrift server?
I enabled spark.sql.thriftServer.incrementalCollect in my Thrift server (Spark 3.1.2) to prevent OutOfMemory exceptions. This worked fine, but my queries are really slow now. I checked the logs and found that Thrift is querying batches of 10.000 rows.
INFO SparkExecuteStatementOperation: Returning result set with 10000 rows from offsets [1260000, 1270000) with 169312d3-1dea-4069-94ba-ec73ac8bef80
My hardware would be able to handle 10x-50x of that.
This issue and this documentation page suggest setting spark.sql.inMemoryColumnarStorage.batchSize, but that didn't work.
Is it possible to configure the value?
Upvotes: 3
Views: 302
Answers (1)
RockSolid
Reputation: 536
The spark.sql.inMemoryColumnarStorage.batchSize is for caching not for the fetchSize per incrementalload. Read the spark thrift code in open source repo to check exact usage.
Upvotes: 0
Related Questions
- How to make spark streaming commit in each batch when limiting Kafka batch size?
- Limit Batch Size in Apache Spark 3.0 Structured Streaming - MicroBatchStream
- How to set batch size in one micro-batch of spark structured streaming
- Spark Error - Max iterations (100) reached for batch Resolution
- Spark Streaming tuning number of records per batch size not working?
- How to set maximum number of rows in one micro-batch?
- How to set the number of documents processed in a batch?
- how to set spark.sql.thriftServer.incrementalCollect=true
- How do you override the Spark Java heap size?
- Specifying the number of partitions for cached data in Spark SQL Thrift Server