CODEWITHSUNDEEP
scala
user3117639
user3117639

Reputation:

Spark Memory issue

Hi I am very new to Spark.I am trying to execute the following command on Apache Spark scala command line

        scala> val files=sc.textFile("/home/test/u.item")
               14/12/06 09:57:40 INFO MemoryStore: ensureFreeSpace(165239) called with 
               curMem=0, maxMem=278302556
               14/12/06 09:57:40 INFO MemoryStore: Block broadcast_0 stored as values in 
               memory (estimated size 161.4 KB, free 265.3 MB)
               files: org.apache.spark.rdd.RDD[String] = /home/test/u.item MappedRDD[1] at
               textFile at :12

May I please know the steps to correct the above error

Upvotes: 2

Views: 2014

Answers (2)

Walter
Walter

Reputation: 397

As MiguelPeralvo said, the level in log4j.properties can be changed to WARN. To do this, go to the conf folder in the spark directory. If log4j.properties doesn't exist, make it from the template in that dir. 

cp log4j.properties.template log4j.properties

then open log4j.properties in an editor and change the first line to 

log4j.rootCategory=WARN, console

restart spark and the INFO messages should be gone.

Upvotes: 0

MiguelPeralvo
MiguelPeralvo

Reputation: 877

This is not an error message. log4j is logging it as an INFO message (see below the INFO string, between the 2 stars). If it was an error, it would have ERROR level (WARN for warnings). Your output is expected when you invoke the textFile method of the context. If you want to log only warnings and errors, I suggest you to change the level in log4j.properties to WARN.

14/12/06 09:57:40 **INFO** MemoryStore: ensureFreeSpace(165239) called with 
           curMem=0, maxMem=278302556
14/12/06 09:57:40 **INFO** MemoryStore: Block broadcast_0 stored as values in 
           memory (estimated size 161.4 KB, free 265.3 MB)

I'm not an expert in Spark, but I guess that ensureFreeSpace is a function invoked by MemoryStore to retrieve memory for the Spark Context's activity.

Upvotes: 2

Related Questions