CODEWITHSUNDEEP
cassandrahivedatastax-enterpriseshark-sql
user3632180
user3632180

Reputation: 105

Shark integration with datastax enterprise 4.0.3 cassandra

I'm trying to integrate Shark 0.9.1 (for hadoop 1) with hive on datastax enterprise 4.0.3 Hadoop node. I have successfully intsall and configure Scala 2.10.3 and Spark 1.0.0. Scala and sparks shells are also working fine. Now when i'm trying open Shark shell with SHARK_HOME/bin/shark then I'm getting error 

 14/06/26 10:04:14 DEBUG metastore.CassandraHiveMetaStore: Creating CassandraHiveMetaStore
 14/06/26 10:04:15 ERROR config.DatabaseDescriptor: Fatal configuration error
    org.apache.cassandra.exceptions.ConfigurationException: Cannot locate cassandra.yaml
    at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:64)
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:75)
    at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:135)
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111)
    at org.apache.cassandra.service.StorageService.getPartitioner(StorageService.java:148)
    at org.apache.cassandra.service.StorageService.<init>(StorageService.java:142)
    at org.apache.cassandra.service.StorageService.<clinit>(StorageService.java:144)
    at com.datastax.bdp.hadoop.hive.metastore.CassandraClientHolder.<init>(CassandraClientHolder.java:69)
    at com.datastax.bdp.hadoop.hive.metastore.SchemaManagerService.<init>(SchemaManagerService.java:111)
    at com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStore.setConf(CassandraHiveMetaStore.java:114)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2136)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2147)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1043)
    at shark.memstore2.TableRecovery$.reloadRdds(TableRecovery.scala:49)
    at shark.SharkCliDriver.<init>(SharkCliDriver.scala:283)
    at shark.SharkCliDriver$.main(SharkCliDriver.scala:162)
    at shark.SharkCliDriver.main(SharkCliDriver.scala)
 Cannot locate cassandra.yaml
  Fatal configuration error; unable to start. See log for stacktrace.

Can anyone faced this kind of issue. Please suggest . Thanks

Upvotes: 0

Views: 215

Answers (1)

Jacek L.
Jacek L.

Reputation: 1416

Try DataStax Enterprise 4.5, it has built-in support for Spark and Shark.

Upvotes: 1

Related Questions