CODEWITHSUNDEEP
scalaapache-sparkdistributed-computing
Amitesh Ranjan
Amitesh Ranjan

Reputation: 1472

spark <console>:12: error: not found: value sc

I wrote the following:

val a = 1 to 10000
val b = sc.parallelize(a)

and it shows error saying:

<console>:12: error: not found: value sc

Any help?

Upvotes: 7

Views: 28221

Answers (7)

Franc Drobnič
Franc Drobnič

Reputation: 1085

As stated in this thread, one solution may be to switch off permissions checking.

In cloudera manager, go to hdfs configuration under advanced and put the following code in "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" :

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

After that, it is necessary to restart the HDFS component.

It worked for me. It might not be appropriate for a production environment, however.

Upvotes: 0

satish sasate
satish sasate

Reputation: 31

In my case I have spark installed on local windows system and I observed the same error but it was because of below issue

Issue:Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

This was because of permission issue.I resolved it by changing the permissions using below command.Though log says "on hdfs" this is on windows system

E:\winutils\bin\winutils.exe chmod 777 E:\tmp\hive

Upvotes: 3

Santosh Gaikwad
Santosh Gaikwad

Reputation: 16

i faced the same problem. In my case the JAVA_HOME was not set properly which cause this issue. surprisingly SPARK would start but the sc context had issues creating an instance. When i fixed the JAVA_HOME to point to the correct java directory, this issue was resolved. I had to close the session and re-open a new one in order to ensure the path is updated and fresh session is turned on.

I hope this helps.

Upvotes: 0

uday
uday

Reputation: 11

First check the log file after spark-shell command run whether SparkContext is initinalized as sc. if SparkContext is not initialized properly

you have to set the IP address in spark environment.

Open the env file in conf/spark.env.sh and add the below line

export SPARK_LOCAL_IP="127.0.0.1"

Upvotes: 1

gsamaras
gsamaras

Reputation: 73366

You get this error, because sc is not defined. I would try:

sc = SparkContext(appName = "foo")

Another thing that usually happens to me is not getting a Kerberos ticket in the cluster, because I forgot too.

As for the "open issue in Spark" mentioned by Solnanki, I am pretty sure this is not the case any more.

Upvotes: 1

user3321437
user3321437

Reputation: 95

I hit this error when trying out Spark on Cloudera Quickstart VM. Turned out to be an hdfs file permission issue on /user/spark.

I could not switch to user "spark", I got a user not available error. Changing file permissions with the below command solved it for me.

sudo -u hdfs hadoop fs -chmod -R 1777 /user/spark

scala> val data = 1 to 10000
data: scala.collection.immutable.Range.Inclusive = Range(1, 2, 3, 4, 5, 6, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170...
scala> val distData = sc.parallelize(data)
distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:14

Upvotes: 0

Shyamendra Solanki
Shyamendra Solanki

Reputation: 8851

It happens when your classpath is not correct. This is an open issue in Spark at the moment.

> spark-shell 

...
...
14/08/08 18:41:50 INFO SparkILoop: Created spark context..
Spark context available as sc.

scala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@2c1c5c2e

scala> :cp /tmp
Added '/tmp'.  Your new classpath is:
...

scala> sc
<console>:8: error: not found: value sc

You may need to correct your classpath from outside the repl.

Upvotes: 2

Related Questions