There is insufficient memory for the Java Runtime Environment to continue with spark-submit

Question

I am running pyspark job

spark-submit --driver-memory 2g --executor-memory 2g  --conf spark.driver.maxResultSize=2g job.py

I tried changing multiple options but every time i am getting below error:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000072324e000, 985616384, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (mmap) failed to map 985616384 bytes for committing reserved memory.

An error report file with more information is saved as:

I am new to spark can some one help me out what should be the solution with this.

Answer 1

Try reducing the driver memory - the node where you're submitting the job is running out of RAM.