Hadoop daemons in a secured cluster fails to start with "Unable to obtain password from user"

Products

Services Suite

Issue/Introduction

Symptoms:

In a secured Hadoop cluster, you can come across situations in which Hadoop daemons (namenode, datanode etc.) fails to start due to Kerberos authentication issues. The error message "Unable to obtain password from user" is produced.

Daemons logs will help you identify investigate the problem further.

The output from the namenode logs are displayed below:

2014-03-27 17:57:57,904 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: Login failure for hdfs/dev6ha@SATURN.LOCAL from keytab /etc/security/phd/keytab/hdfs.service.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:836)
..
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:789)
..
2014-03-27 18:15:33,186 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-03-27 18:15:33,188 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hdm1.saturn.local/10.246.67.243

Environment

Cause

The cause of this issue is incorrect configuration.

Resolution

The instructions below will help you start the investigation. We have used the above log snippet values for this example.

Identify what is the principal name?

: hdfs/dev6ha@SATURN.LOCAL

The Kerberos principal name is specified in hdfs-site.xml. It is used to authenticate against Kerberos.
Verify if the parameter dfs.namenode.kerberos.*.principal / dfs.datanode.kerberos.*.principal has correct principal name associated with the node role.
If using the _HOST variable in hdfs-site.xml, ensure that hostname -f is returning the fully qualified hostname and it matches with the principal name as indicated in the log files.
If you have configuration issues, it will not replace _HOST with the correct name and you may see errors. For example DNS returns an IP but in /etc/hosts there is a different IP specified for the same hosts.

Note: In this example, DNS returned the IP as 10.246.67.243, but /etc/hosts is pointing to 10.246.67.218. _HOST is getting replaced by the nameservice name (dev6ha) instead of the actual hostname because this was a NameNode High Availability configuration.

Identify what is the keytab file used?

/etc/security/

/etc/security/phd/keytab/hdfs.service.keytab

If the keytab file defined in hdfs-site.xml is not present, an error will be produced. Verify the path and the keytab filename.

Verify if you can kinit using the principal name and keytab?

[root@phd11-nn keytab] kinit -ket /etc/security/phd/keytab/hdfs.service.keytab hdfs/[email protected]

If kinit is failing then there might be a problem with the hostname IP mapping in your keytab file that are inconsistent with DNS or /etc/hosts, and you can still get the same error.

Execute the following command to verify contents of the keytab file:

klist -ket /etc/security/phd/keytab/hdfs.server.keytab

Execute the following command to regenerate the keytab file:

[root@KDC server] kadmin.local
ktadd -norandkey -k /etc/security/keytab/hdfs-hostid.service.keytab  hdfs/host_fqdn@REALM  HTTP/host_fqdn@REALM