CODEWITHSUNDEEP
hadoophdfskerberos
Ururu
Ururu

Reputation: 13

HDFS AuthorizationException

Please help me to understand why I have problem with datanode connection below:

WARN server.AuthenticationFilter (AuthenticationFilter.java:doFilter(588)) - Authentication exception: org.apache.hadoop.security.authentication.client.AuthenticationException: GSS Exception: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(227)) - Problem connecting to server: s--t-..ru/10.243..*:8020

I have kerberized cluster and everything works fine but I need to add new datanode and I have connection problem only with new datanode. On namenode I have next messages:

INFO ipc.Server (Server.java:authorizeConnection(2235)) - Connection from 10.243.218.16:33435 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/s--t-..ru@.RU (auth:PROXY) via $J4LB00-3PQ0LQ7EGVSG@.RU (auth:KERBEROS) 2020-02-05 09:37:20,172 INFO ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client 10.243.218.16 threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $J4LB00-3PQ0LQ7EGVSG@.RU is not allowed to impersonate dn/s--t-..ru@.RU]

So what the most interesting things it is a User: $J4LB00-3PQ0LQ7EGVSG@.RU it the same dn/s--t-..ru user but pre-win 2000 logon name

Is it right things? What else interesting i don't have problem with other elder datanode only with this one.

Upvotes: 1

Views: 3287

Answers (3)

Richard Zuidhof
Richard Zuidhof

Reputation: 11

Thank you for reporting this. I investigated the issue on a HDP 3.1.4 cluster for a few hours but could not grasp what was going on. Downgrading JDK fixed it.

This OpenJDK 8 242 release is causing more issues. We had to rollback to OpenJDK 8u232 for HDF (Nifi) as well for an issue with Nifi Site-to-site connections giving timeouts, not Kerberos related. Symptom was that TCP sessions to the raw port were in state SYN_RECV.

Upvotes: 1

Andreas Ufert
Andreas Ufert

Reputation: 11

Downgrading is not a real fix. We have been hit by the same problem so I reported it here: https://bugs.launchpad.net/ubuntu/+source/openjdk-8/+bug/1861883

I also reported this to the OpenJDK bug report site (twice) but no reaction so far.

Upvotes: 0

Christian Weiß
Christian Weiß

Reputation: 46

We ran into exactly the same issue this week on our cluster, affecting all nodes!

In HDP 3.1.4 release notes on page 72 we found a known issue regarding Open JDK 8u242:

Description of the problem or behavior Open JDK 8u242 is not supported as it causes Kerberos failure. Workaround Use a different version of Open JDK

We figured out that we did an upgrade to 8u242 in the beginning of the week and downgraded to openjdk-8-jdk=8u162-b12-1.

That fixed the problem for us.

Upvotes: 3

Related Questions