Fails to access HDFS secured with Kerberos
Article ID: 295104
Updated On:
Products
Issue/Introduction
Attempting to access kerberized Hadoop Distributed File System (HDFS) on any cluster host fails with the error message "SIMPLE authentication is not enabled" even though all Kerberos parameters are configured correctly.
# hdfs dfs -ls / ls: Failed on local exception: java.io.IOException: Couldn't setup connection for [email protected] to node01.example.com/172.28.21.2:8020; Host Details : local host is: "node01.example.com/172.28.21.2"; destination host is: "node01.example.com":8020;
The output below is from the namenode logs:
2015-11-25 16:15:34,792 INFO ipc.Server (Server.java:doRead(771)) - Socket Reader #1 for port 8020: readAndProcess from client 172.28.21.2 threw exception [org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]] org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Server$Connection.initializeAuthContext(Server.java:1554) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1510) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:762) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:636) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607)
Environment
Cause
This issue is caused by a large clock skew between the KDC server and the servers in the Hadoop cluster.
RCA
Enable HADOOP DEBUG and run the Hadoop client command again to get more debugging logs. Confirm that the logs show the error message "Clock skew too great".
# HADOOP_ROOT_LOGGER=DEBUG,console hdfs dfs -ls / ...... 15/11/26 13:15:54 DEBUG security.UserGroupInformation: hadoop login 15/11/26 13:15:54 DEBUG security.UserGroupInformation: hadoop login commit 15/11/26 13:15:54 DEBUG security.UserGroupInformation: using kerberos user:[email protected] 15/11/26 13:15:54 DEBUG security.UserGroupInformation: Using user: "[email protected]" with name [email protected] 15/11/26 13:15:54 DEBUG security.UserGroupInformation: User entry: "[email protected]" 15/11/26 13:15:54 DEBUG security.UserGroupInformation: UGI loginUser:[email protected] (auth:KERBEROS) ...... 15/11/26 13:16:11 DEBUG security.UserGroupInformation: PrivilegedActionException as:[email protected] (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)] 15/11/26 13:16:11 DEBUG security.UserGroupInformation: PrivilegedAction as:[email protected] (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:644) 15/11/26 13:16:11 WARN ipc.Client: Couldn't setup connection for [email protected] to node01.example.com/172.28.21.2:8020 15/11/26 13:16:11 DEBUG security.UserGroupInformation: PrivilegedActionException as:[email protected] (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for [email protected] to node01.example.com/172.28.21.2:8020 15/11/26 13:16:11 DEBUG ipc.Client: closing ipc connection to node01.example.com/172.28.21.2:8020: Couldn't setup connection for [email protected] to node01.example.com/172.28.21.2:8020 java.io.IOException: Couldn't setup connection for [email protected] to node01.example.com/172.28.21.2:8020 ...... at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
Observe that the system time is not synchronized between the KDC server and the servers on the Hadoop cluster.
System time on KDC server:
# date Thu Nov 26 05:20:54 GMT+8 2015
System time on Hadoop cluster server:
# date Thu Nov 26 13:20:41 SGT 2015
Run kinit to successfully receive a Kerberos credential.
Observe that the valid period of the Kerberos credential is far ahead of the current system time. As a result, the credential cannot be used immediately to access HDFS.
# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: [email protected] Valid starting Expires Service principal 11/27/15 06:28:31 11/27/15 14:28:31 krbtgt/[email protected] # date Thu Nov 26 14:29:19 SGT 2015
Resolution
To resolve this issue, follow the instructions below:
1. Use ntp to synchronize the system time of the KDC server with the servers in Hadoop cluster.
2. Confirm that the UTC times on all servers (KDC and Hadoop cluster servers) are synchronized. This will determine a valid period of Kerberos credentials. In fact, KDC will set a valid period of Kerberos credential based on UTC time and the Kerberos client will adjust it according to timezone.
Feedback
