CODEWITHSUNDEEP
hadoopapache-kafkaapache-stormkerberoshadoop-streaming
Niraj
Niraj

Reputation: 51

Facing Issue in Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster

I have enabled Kerberos Security in my HDP 2.6 cluster, on which services- Kafka and Storm are installed prior to enabling Kerberos.

The topology that I am running has kafka-spout followed by hdfs-bolt. So, CSV data from a specific Kafka topic is ingested using in-built Kafka Spout, and then transferred to HDFS directory using in-built HDFS Bolt.

I am Facing an error: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

Code:

Map<String, Object> map = new HashMap<String,Object>();
    map.put("hdfs.keytab.file","/etc/security/keytabs/storm.headless.keytab");
    map.put("hdfs.kerberos.principal","[email protected]");
    Config conf = new Config();
    conf.put("hdfs.config", map);
    conf.put(HdfsSecurityUtil.STORM_KEYTAB_FILE_KEY,"/etc/security/keytabs/storm.headless.keytab");
    conf.put(HdfsSecurityUtil.STORM_USER_NAME_KEY,"[email protected]");
    conf.setNumWorkers(1);

    HdfsBolt hdfsbolt = new HdfsBolt()
            .withFsUrl(hdfsUrl)
            .withFileNameFormat(fileNameFormat).withRecordFormat(recordFormat)
            .withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy).withConfigKey("hdfs.config");

StormSubmitter.submitTopology(topologyName, conf, builder.createTopology());

Error:

Thread-7-hdfs-bolt-executor[3 3] [ERROR]
org.apache.hadoop.security.AccessControlException: 
SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]

Please let me know if I need to do any other steps/config related changes for storm-hdfs connector to work with Kerberos.

Upvotes: 1

Views: 238

Answers (1)

VenkataAvinash Kesineni
VenkataAvinash Kesineni

Reputation: 1

According to the mentioned code you are using withConfigKey method from hdfsBolt Adding topology.classpath: "/etc/hadoop/conf" in storm.yaml solved the above issue in my case

Upvotes: 0

Related Questions