CODEWITHSUNDEEP
mavenapache-spark
BlueSheepToken
BlueSheepToken

Reputation: 6169

Maven seems ignoring MAVEN_OPTS

I am actually working with spark and I am trying to increase the heapspace.

I have read several topics and followed them.

I am working on windows and I have set the variable environment : MAVEN_OPTS=-Xmx512m with set MAVEN_OPTS=-Xmx512m and checked it with set M

However, what ever the Xmx value is set to. I have the same error.

 Caused by: java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.

Is there a way to check what the maximum heap space of the jvm is? Or is there a work around while instantiating the spark session? Such as SparkSession.builder.config("spark.driver.memory", "512m") ?

I am actually running maven from the command line mvn test on Windows. The tests instantiate a sparkSession which results in the error

As I am coding in scala, for the test plugin I have this :

        <plugin>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest-maven-plugin</artifactId>
            <version>2.0.0</version>
            <executions>
                <execution>
                    <id>test</id>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

Upvotes: 2

Views: 1302

Answers (1)

Little Santi
Little Santi

Reputation: 8813

In the case the cause of your overflow problem was that the scalatest-maven-plugin was forking another JVM, I suggest you include the argLine property with the proper -X options:

<plugin>
    <groupId>org.scalatest</groupId>
    <artifactId>scalatest-maven-plugin</artifactId>
    <version>2.0.0</version>
    <configuration>
        <argLine>-Xmx512m</argLine>
        ...
    </configuration>
    ...
<plugin>

See scalatest-maven-plugin.

Upvotes: 4

Related Questions