CODEWITHSUNDEEP
hadoopapache-sparkhivepysparkapache-spark-sql
Arnab
Arnab

Reputation: 1037

Installing Hive with Apache Spark

I was trying to execute the following query in Spark:

from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")

But this resulted in the error:

File "<stdin>", line 1, in <module>

File "/home/hduser/Software/spark/python/pyspark/sql/context.py", line 502, in sql
return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
 File "/home/hduser/Software/spark/python/pyspark/sql/context.py", line 610, in _ssql_ctx
"build/sbt assembly", e)
Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly", Py4JError(u'Trying to call a package.',))

I was trying to run a Hive query on top of Spark. Is it necessary to build Spark with Hive, I already have Spark and Hive installed independently on my system, is there a way by which I can run Hive queries on spark without the need to build spark using my existing configurations.

Thanks in advance

Below is the complete log file:

16/01/07 02:50:24 DEBUG PythonGatewayServer: Started PythonGatewayServer on port 53473
16/01/07 02:50:24 DEBUG PythonGatewayServer: Communicating GatewayServer port to Python driver at 127.0.0.1:48570
16/01/07 02:50:24 INFO SparkContext: Running Spark version 1.4.1
16/01/07 02:50:24 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of successful kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
16/01/07 02:50:24 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of failed kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
16/01/07 02:50:24 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[GetGroups], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
16/01/07 02:50:24 DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
16/01/07 02:50:24 DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
    at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:303)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:328)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:610)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2162)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2162)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2162)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:301)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)
16/01/07 02:50:24 DEBUG Shell: setsid exited with exit code 0
16/01/07 02:50:24 DEBUG KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
16/01/07 02:50:24 DEBUG Groups:  Creating new Groups object
16/01/07 02:50:24 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
16/01/07 02:50:24 DEBUG NativeCodeLoader: Loaded the native-hadoop library
16/01/07 02:50:24 DEBUG JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
16/01/07 02:50:24 DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
16/01/07 02:50:24 DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
16/01/07 02:50:24 DEBUG UserGroupInformation: hadoop login
16/01/07 02:50:24 DEBUG UserGroupInformation: hadoop login commit
16/01/07 02:50:24 DEBUG UserGroupInformation: using local user:UnixPrincipal: hduser
16/01/07 02:50:24 DEBUG UserGroupInformation: Using user: "UnixPrincipal: hduser" with name hduser
16/01/07 02:50:24 DEBUG UserGroupInformation: User entry: "hduser"
16/01/07 02:50:24 DEBUG UserGroupInformation: UGI loginUser:hduser (auth:SIMPLE)
16/01/07 02:50:24 WARN SparkConf: 
SPARK_CLASSPATH was detected (set to '/home/hduser/mysql-connector-java-5.1.36-bin.jar').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --driver-class-path to augment the driver classpath
 - spark.executor.extraClassPath to augment the executor classpath

16/01/07 02:50:24 WARN SparkConf: Setting 'spark.executor.extraClassPath' to '/home/hduser/mysql-connector-java-5.1.36-bin.jar' as a work-around.
16/01/07 02:50:24 WARN SparkConf: Setting 'spark.driver.extraClassPath' to '/home/hduser/mysql-connector-java-5.1.36-bin.jar' as a work-around.
16/01/07 02:50:24 WARN Utils: Your hostname, desktop1 resolves to a loopback address: 127.0.1.1; using 192.168.1.101 instead (on interface wlan0)
16/01/07 02:50:24 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
16/01/07 02:50:25 INFO SecurityManager: Changing view acls to: hduser
16/01/07 02:50:25 INFO SecurityManager: Changing modify acls to: hduser
16/01/07 02:50:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser)
16/01/07 02:50:25 DEBUG SSLOptions: No SSL protocol specified
16/01/07 02:50:25 DEBUG SSLOptions: No SSL protocol specified
16/01/07 02:50:25 DEBUG SSLOptions: No SSL protocol specified
16/01/07 02:50:25 DEBUG SecurityManager: SSLConfiguration for file server: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
16/01/07 02:50:25 DEBUG SecurityManager: SSLConfiguration for Akka: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
16/01/07 02:50:25 DEBUG AkkaUtils: In createActorSystem, requireCookie is: off
16/01/07 02:50:25 INFO Slf4jLogger: Slf4jLogger started
16/01/07 02:50:25 INFO Remoting: Starting remoting
16/01/07 02:50:26 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:36696]
16/01/07 02:50:26 INFO Utils: Successfully started service 'sparkDriver' on port 36696.
16/01/07 02:50:26 DEBUG SparkEnv: Using serializer: class org.apache.spark.serializer.JavaSerializer
16/01/07 02:50:26 INFO SparkEnv: Registering MapOutputTracker
16/01/07 02:50:26 INFO SparkEnv: Registering BlockManagerMaster
16/01/07 02:50:26 INFO DiskBlockManager: Created local directory at /tmp/spark-10be872e-6114-4f74-9546-7ea87fd03425/blockmgr-adcc8ff0-29d5-4168-904b-38f822d38186
16/01/07 02:50:26 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
16/01/07 02:50:26 INFO HttpFileServer: HTTP File server directory is /tmp/spark-10be872e-6114-4f74-9546-7ea87fd03425/httpd-1112dc78-2447-4bb9-86f6-3c2c725b0951
16/01/07 02:50:26 INFO HttpServer: Starting HTTP Server
16/01/07 02:50:26 DEBUG HttpServer: HttpServer is not using security
16/01/07 02:50:26 INFO Utils: Successfully started service 'HTTP file server' on port 42190.
16/01/07 02:50:26 DEBUG HttpFileServer: HTTP file server started at: http://192.168.1.101:42190
16/01/07 02:50:26 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/07 02:50:26 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/01/07 02:50:26 INFO SparkUI: Started SparkUI at http://192.168.1.101:4040
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(ExpireDeadHosts,false) from Actor[akka://sparkDriver/deadLetters]
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(ExpireDeadHosts,false)
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (5.176845 ms) AkkaMessage(ExpireDeadHosts,false) from Actor[akka://sparkDriver/deadLetters]
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(TaskSchedulerIsSet,false) from Actor[akka://sparkDriver/deadLetters]
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(TaskSchedulerIsSet,false)
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.232258 ms) AkkaMessage(TaskSchedulerIsSet,false) from Actor[akka://sparkDriver/deadLetters]
16/01/07 02:50:26 INFO Executor: Starting executor ID driver on host localhost
16/01/07 02:50:26 DEBUG InternalLoggerFactory: Using SLF4J as the default logging framework
16/01/07 02:50:26 DEBUG PlatformDependent0: java.nio.Buffer.address: available
16/01/07 02:50:26 DEBUG PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
16/01/07 02:50:26 DEBUG PlatformDependent0: sun.misc.Unsafe.copyMemory: available
16/01/07 02:50:26 DEBUG PlatformDependent0: java.nio.Bits.unaligned: true
16/01/07 02:50:26 DEBUG PlatformDependent: UID: 1001
16/01/07 02:50:26 DEBUG PlatformDependent: Java version: 7
16/01/07 02:50:26 DEBUG PlatformDependent: -Dio.netty.noUnsafe: false
16/01/07 02:50:26 DEBUG PlatformDependent: sun.misc.Unsafe: available
16/01/07 02:50:26 DEBUG PlatformDependent: -Dio.netty.noJavassist: false
16/01/07 02:50:26 DEBUG PlatformDependent: Javassist: unavailable
16/01/07 02:50:26 DEBUG PlatformDependent: You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes.  Please check the configuration for better performance.
16/01/07 02:50:26 DEBUG PlatformDependent: -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
16/01/07 02:50:26 DEBUG PlatformDependent: -Dio.netty.bitMode: 64 (sun.arch.data.model)
16/01/07 02:50:26 DEBUG PlatformDependent: -Dio.netty.noPreferDirect: false
16/01/07 02:50:26 DEBUG MultithreadEventLoopGroup: -Dio.netty.eventLoopThreads: 8
16/01/07 02:50:26 DEBUG NioEventLoop: -Dio.netty.noKeySetOptimization: false
16/01/07 02:50:26 DEBUG NioEventLoop: -Dio.netty.selectorAutoRebuildThreshold: 512
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numHeapArenas: 4
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numDirectArenas: 4
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.pageSize: 8192
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxOrder: 11
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.chunkSize: 16777216
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.tinyCacheSize: 512
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.smallCacheSize: 256
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.normalCacheSize: 64
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxCachedBufferCapacity: 32768
16/01/07 02:50:26 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.cacheTrimInterval: 8192
16/01/07 02:50:26 DEBUG ThreadLocalRandom: -Dio.netty.initialSeedUniquifier: 0x09852a335e6ac767 (took 0 ms)
16/01/07 02:50:26 DEBUG ByteBufUtil: -Dio.netty.allocator.type: unpooled
16/01/07 02:50:26 DEBUG ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 65536
16/01/07 02:50:26 DEBUG NetUtil: Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%1)
16/01/07 02:50:26 DEBUG NetUtil: /proc/sys/net/core/somaxconn: 128
16/01/07 02:50:26 DEBUG TransportServer: Shuffle server started on port :40229
16/01/07 02:50:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40229.
16/01/07 02:50:26 INFO NettyBlockTransferService: Server created on 40229
16/01/07 02:50:26 INFO BlockManagerMaster: Trying to register BlockManager
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(RegisterBlockManager(BlockManagerId(driver, localhost, 40229),278302556,AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/BlockManagerEndpoint1#-2004000522])),true) from Actor[akka://sparkDriver/temp/$a]
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(RegisterBlockManager(BlockManagerId(driver, localhost, 40229),278302556,AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/BlockManagerEndpoint1#-2004000522])),true)
16/01/07 02:50:26 INFO BlockManagerMasterEndpoint: Registering block manager localhost:40229 with 265.4 MB RAM, BlockManagerId(driver, localhost, 40229)
16/01/07 02:50:26 INFO BlockManagerMaster: Registered BlockManager
16/01/07 02:50:26 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (10.023007 ms) AkkaMessage(RegisterBlockManager(BlockManagerId(driver, localhost, 40229),278302556,AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/BlockManagerEndpoint1#-2004000522])),true) from Actor[akka://sparkDriver/temp/$a]
16/01/07 02:50:43 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@25ad7569,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$b]
16/01/07 02:50:43 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@25ad7569,BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:50:43 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (2.348068 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@25ad7569,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$b]
16/01/07 02:50:43 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$c]
16/01/07 02:50:43 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:50:43 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (6.460111 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$c]
16/01/07 02:50:53 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@1bd046bb,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$d]
16/01/07 02:50:53 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@1bd046bb,BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:50:53 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (1.269133 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@1bd046bb,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$d]
16/01/07 02:50:53 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$e]
16/01/07 02:50:53 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:50:53 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (1.066398 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$e]
16/01/07 02:51:03 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@47c2b6e3,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$f]
16/01/07 02:51:03 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@47c2b6e3,BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:51:03 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.933259 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@47c2b6e3,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$f]
16/01/07 02:51:03 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$g]
16/01/07 02:51:03 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:51:03 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (1.05242 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$g]
16/01/07 02:51:13 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@407515e3,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$h]
16/01/07 02:51:13 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@407515e3,BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:51:13 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (1.032808 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;@407515e3,BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$h]
16/01/07 02:51:13 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$i]
16/01/07 02:51:13 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true)
16/01/07 02:51:13 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (1.713425 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 40229)),true) from Actor[akka://sparkDriver/temp/$i]

Upvotes: 0

Views: 3514

Answers (1)

Brian Clapper
Brian Clapper

Reputation: 26230

It is trivial to build Spark locally with Hive support, but it does not have Hive support out of the box (because of the large number of dependencies Hive pulls in). Here's the command line I usually use to build Spark, from source, to include Hive support:

./make-distribution.sh --name spark-hive-1.5.2 --tgz -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver

Obviously, I'm building Spark 1.5.2, but the above should work for any version. The --name parameter just allows me to name the final built distribution.

See also: https://spark.apache.org/docs/latest/building-spark.html#building-with-hive-and-jdbc-support

Please note that the versions you can download from spark.apache.org don't have Hive support (as noted in the above URL).

Upvotes: 3

Related Questions