CODEWITHSUNDEEP
hivekerberoshortonworks-data-platformhue
Arnaud
Arnaud

Reputation: 273

Hue Beeswax / HCat no longer working (kerberos default user) after migration to HDP2.2

I've almost done the migration of my secure HDP2.1 to HDP2.2 hadoop cluster. Everything seems to work (including hive in command line), but hue. If the file browser, job browser, pig interface and oozie interface are working, this is not the case of the beeswax & webhcat interface. (NB : they were working before the migration, with the same hue.ini file).

The error I get is : Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database)

It seems that thrift is trying to authenticate a default user krbtgt/LOCALDOMAIN instead of the configured ones.

I've tried to log what happens in the python file but failed to see where it gets that default user : kerberos principal short name is hive, impersonification is enabled. Hue & hive proxies are configured in the hdfs conf files.

The complete stack trace is :

[11/May/2015 06:10:40 +0000] access       INFO     172.20.43.39 alinz - "GET /beeswax/ HTTP/1.0"
[11/May/2015 06:10:40 +0000] hive_server2_lib INFO     use_sasl=True, mechanism=GSSAPI, kerberos_principal_short_name=hive, impersonation_enabled=True
[11/May/2015 06:10:40 +0000] thrift_util  INFO     Thrift exception; retrying: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database)
[11/May/2015 06:10:40 +0000] thrift_util  INFO     Thrift exception; retrying: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database)
[11/May/2015 06:10:40 +0000] thrift_util  WARNING  Out of retries for thrift call: OpenSession
[11/May/2015 06:10:40 +0000] thrift_util  INFO     Thrift saw a transport exception: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database)
[11/May/2015 06:10:40 +0000] middleware   INFO     Processing exception: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database) (code THRIFTTRANSPORT): TTransportException('Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database)',): Traceback (most recent call last):
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 69, in index
    return execute_query(request)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 526, in execute_query
    databases = _get_db_choices(request)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 1849, in _get_db_choices
    dbs = _get_databases(request)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/views.py", line 1844, in _get_databases
    dbs = db.get_databases()
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 110, in get_databases
    return self.client.get_databases()
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 746, in get_databases
    return [table[col] for table in self._client.get_databases()]
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 445, in get_databases
    res = self.call(self._client.GetSchemas, req)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 408, in call
    session = self.open_session(self.user)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 382, in open_session
    res = self._client.OpenSession(req)
  File "/usr/lib/hue/desktop/core/src/desktop/lib/thrift_util.py", line 329, in wrapper
    raise StructuredThriftTransportException(e, error_code=502)
StructuredThriftTransportException: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database) (code THRIFTTRANSPORT): TTransportException('Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[email protected] not found in Kerberos database)',)

Any idea what could be wrong?

krb5.conf is :


    [libdefaults]
      renew_lifetime = 7d
      forwardable = true
      default_realm = HADOOP.DEV
      ticket_lifetime = 24h
      dns_lookup_realm = false
      dns_lookup_kdc = false
    [logging]
      default = FILE:/var/log/krb5kdc.log
      admin_server = FILE:/var/log/kadmind.log
      kdc = FILE:/var/log/krb5kdc.log
    [realms]
      HADOOP.DEV = {
        admin_server = bt1svlmy
        kdc = bt1svlmy
      }

and sudo klist -e /tmp/hue_krb5_ccache gives:

Ticket cache: FILE:/tmp/hue_krb5_ccache
Default principal: hue/[email protected]

Valid starting     Expires            Service principal
05/11/15 15:10:34  05/12/15 15:10:34  krbtgt/[email protected]
        renew until 05/11/15 15:10:34, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
05/11/15 15:49:52  05/12/15 15:10:34  HTTP/bt1svlmy.bpa.bouyguestelecom.fr@
        renew until 05/11/15 15:10:34, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
05/11/15 15:49:52  05/12/15 15:10:34  HTTP/[email protected]
        renew until 05/11/15 15:10:34, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

I do have a krbtgt/[email protected]ticket but no krbtgt/[email protected] ; maybe it's the cause of the issue?

Kerberos log file is :

May 11 16:12:35 bt1svlmy krb5kdc[12636](info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0,  hue/[email protected] for hive/[email protected], Server not found in Kerberos database
May 11 16:12:35 bt1svlmy krb5kdc[12636](info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0,  hue/[email protected] for krbtgt/[email protected], Server not found in Kerberos database
May 11 16:12:35 bt1svlmy krb5kdc[12636](info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0,  hue/[email protected] for hive/[email protected], Server not found in Kerberos database
May 11 16:12:35 bt1svlmy krb5kdc[12636](info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0,  hue/[email protected] for krbtgt/[email protected], Server not found in Kerberos database
May 11 16:12:35 bt1svlmy krb5kdc[12636](info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0,  hue/[email protected] for hive/[email protected], Server not found in Kerberos database
May 11 16:12:35 bt1svlmy krb5kdc[12636](info): TGS_REQ (4 etypes {18 17 16 23}) 172.19.115.50: UNKNOWN_SERVER: authtime 0,  hue/[email protected] for krbtgt/[email protected], Server not found in Kerberos database

It seems to me that I missed a default hostname in the conf somewhere, but could not find the documentation entry for it.

Upvotes: 3

Views: 1575

Answers (1)

Arnaud
Arnaud

Reputation: 273

Okay, found it (had to debug the full python stack to understand). It's not really advertised, but some hue.ini parameter names have changed:

  • beeswax_server_host --> hive_server_host
  • beeswax_server_port --> hive_server_port

It was defaulting hive_server_host to localhost, which is not correct on a secure cluster.

Upvotes: 2

Related Questions