CODEWITHSUNDEEP
hadoophdfswebhdfs
Wei
Wei

Reputation: 11

Hadoop webhdfs in secure mode with user not authenticated by filter error

I have configured Hadoop HDFS with security enabled using Kerberos (CDH4.3). It works fine, I can do all the operations using command line tools. There are some other non hadoop cluster nodes need to access HDFS file system thru REST API, so I have webhdfs enabled with Kerberos configured.

My secure webhdfs works fine for operations that don't involve temporary redirect: ie LISTSTATUS, create directory, remove directory, remove files, all work fine. But for two step operations like create file, open file, it failed with below info:

[DEV][root@namenode ~]# curl -i --negotiate -u :   -X     
PUT "http: //namenode:50070/webhdfs/v1/tmp/hosts.txt?op=CREATE"

HTTP/1.1 401
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 0
Server: Jetty(6.1.26.cloudera.2)

HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Set-Cookie:       
hadoop.auth="u=hdfs&p=hdfs@UNIXKRB&t=kerberos&e=1398429948267&s=XhEp/tfs5Pfp04Dp
9yy1moFPnEo=";Path=/
Location: http: //datanode.net:1006/webhdfs/v1/tmp/hosts.txt?      
op=CREATE&delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2BSR7K
hveMEo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA&namenoderpcaddress=
namenode.net:8020&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.cloudera.2)


[DEV][root@namenode~]# curl -i -X PUT -T hosts.txt     
http: //datanode.net:1006/webhdfs/v1/tmp/hosts.txt? 
op=CREATE&delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2
BSR7KhveMEo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA&namenoderpcaddress=
namenode.net:8020&overwrite=false
[2] 19047
[3] 19048
[4] 19049
[DEV][root@namenode~]# HTTP/1.1 100 Continue

HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Expires: Fri, 25 Apr 2014 02:46:26 GMT
Date: Fri, 25 Apr 2014 02:46:26 GMT
Pragma: no-cache
Expires: Fri, 25 Apr 2014 02:46:26 GMT
Date: Fri, 25 Apr 2014 02:46:26 GMT
Pragma: no-cache
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26.cloudera.2)

{"RemoteException":       
{"exception":"SecurityException","javaClassName":"java.lang.SecurityException",
"message":"Failed to obtain user group information: java.io.IOException: 
 Security enabled but user not authenticated by filter"}}
[2]   Done                    curl -i -X PUT -T hosts.txt     
http://datanode.net:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE
[3]   Done                        
delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2BSR7KhveM
Eo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA
[4]-  Done                    namenoderpcaddress=namenode.net:8020
[DEV][root@namenode ~]#

Is there anyone who can shed some lights?

Thanks a lot,

Upvotes: 1

Views: 3267

Answers (1)

PHDFanBoy
PHDFanBoy

Reputation: 21

In my case it was working fine as below:

$ curl -i --negotiate -u : -X PUT "http://nn30.my.dom:50070/webhdfs/v1/tmp/hosts.txt?op=CREATE"
HTTP/1.1 401
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 0
Server: Jetty(7.6.10.v20130312)


HTTP/1.1 307 Temporary Redirect
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Cache-Control: no-cache
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Set-Cookie: hadoop.auth="u=gpadmin&[email protected]&t=kerberos&e=1398897564735&s=p8haj7KXAsUqj5A2WzoE5VxJYA8=";Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: http ://dn33.my.dom:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE&delegation=IgAHZ3BhZG1pbgdncGFkbWluAIoBRbKli0OKAUXWsg9DAxIUt8i6BG2xt5V4wB4vPpXSPm8MMVMSV0VCSERGUyBkZWxlZ2F0aW9uEzE5Mi4xNjguMTAuMzA6NTAwNzA&namenoderpcaddress=nn30.my.dom:8020&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(7.6.10.v20130312)


$ curl -i -X PUT -T hosts.txt "http://dn33.my.dom:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE&delegation=IgAHZ3BhZG1pbgdncGFkbWluAIoBRbKli0OKAUXWsg9DAxIUt8i6BG2xt5V4wB4vPpXSPm8MMVMSV0VCSERGUyBkZWxlZ2F0aW9uEzE5Mi4xNjguMTAuMzA6NTAwNzA&namenoderpcaddress=nn30.my.dom:8020&overwrite=false"
HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Expires: Wed, 30 Apr 2014 12:42:24 GMT
Date: Wed, 30 Apr 2014 12:42:24 GMT
Pragma: no-cache
Cache-Control: no-cache
Expires: Wed, 30 Apr 2014 12:42:24 GMT
Date: Wed, 30 Apr 2014 12:42:24 GMT
Pragma: no-cache
Location: webhdfs ://nn30.my.dom:50070/tmp/hosts.txt
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(7.6.10.v20130312)

NOTE - if your screen capture was not the edited/modified you had few syntax issues ("") and typo things. I bet you missed "" for the http input from first command. check here for your info - http ://hadoop.apache.org/docs/r1.0.4/webhdfs.html

BTW, I used Pivotal HD, there shouldn't be that much difference though.

Upvotes: 1

Related Questions