How to automatically authenticate with Hadoop using Active Directory?

Question

We have an application that accesses Hadoop via HDFS, YARN, and Hive interfaces. This application works fine against Kerberos-secured clusters if kinit has been run. It also works fine if we call UserGroupInformation.loginUserFromKeytab(). We are able to delegate the HDFS and Hive tokens to YARN applications. The thing we cannot figure out is the following scenario:

• The Hadoop cluster is secured using Kerberos
• The Hadoop cluster either uses Active Directory as its KDC, or has established a one-way trust between its KDC and the AD controller.
• Our software is running in a session that has been authenticated using AD directly on Windows, or via PAM or LDAP (or some other mechanism) on Linux.
• Our software queries the active AD session to extract a TGT or equivalent, and relays that information to the Hadoop APIs (via UserGroupInformation, presumably).
• Hadoop authentication is thus achieved without the need for the user to enter a principal, password, or keytab.

We know this is possible in theory, because there are two examples of software that achieve this. The first is HDFS Explorer from RedGate. The second is Hue. However, we just can't seem to figure out the right incantation, and even Hortonworks support can't seem to help.

Answer 1

Hue comes with a LDAP backend that can transparently authenticate users against your company directory,

Hue also comes with a KT renewer command for keeping its Kerberos ticket up to date. It is even ran automatically when using CM.