CODEWITHSUNDEEP
sqlapache-sparkout-of-memorysnappydata
IFH
IFH

Reputation: 161

Increase Java Memory on Spark for Building Large Hash Relations

I am currently trying to run a TPC-H query on SnappyData. At first the query gave me an error saying

ERROR 38000: (SQLState=38000 Severity=-1) (Server=localhost[1528],Thread[DRDAConnThread_29,5,gemfirexd.daemons]) The exception 'Both sides of this join are outside the broadcasting threshold and computing it could be prohibitively expensive. To explicitly enable it, please set spark.sql.crossJoin.enabled = true;' was thrown while evaluating an expression.

After enabling spark's sql crossjoins and re-running the query, the error pop out saying:

java.lang.RuntimeException: Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes
    at org.apache.spark.sql.execution.joins.HashedRelationCache$.get(LocalJoin.scala:621)
    at org.apache.spark.sql.execution.joins.HashedRelationCache.get(LocalJoin.scala)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.init(Unknown Source)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8.apply(WholeStageCodegenExec.scala:367)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8.apply(WholeStageCodegenExec.scala:364)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$25.apply(RDD.scala:820)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$25.apply(RDD.scala:820)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory(HashedRelation.scala:414)
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.init(HashedRelation.scala:424)
    at org.apache.spark.sql.ex

Please let me know how to increase the amount of memory for building hash relations.

Just in case, the query is below and I am trying to run it on a 1GB dataset (I did try the query on an empty dataset and it does work). TPC-H Query 16:

SELECT i_name,
       substr(i_data, 1, 3) AS brand,
       i_price,
       count(DISTINCT (pmod((s_w_id * s_i_id),10000))) AS supplier_cnt
FROM stock,
     item
WHERE i_id = s_i_id
  AND i_data NOT LIKE 'zz%'
  AND (pmod((s_w_id * s_i_id),10000) NOT IN
    (SELECT su_suppkey
     FROM supplier
     WHERE su_comment LIKE '%bad%'))
GROUP BY i_name,
         substr(i_data, 1, 3),
         i_price
ORDER BY supplier_cnt DESC;

Upvotes: 0

Views: 1751

Answers (1)

Kishor Bachhav
Kishor Bachhav

Reputation: 181

In configuration file for server (conf/servers) set jvm memory as -J-Xms5g So conf/server will look like localhost -locators=localhost:10334 -J-Xms5g

Upvotes: 2

Related Questions