Namenode commands fails with "Disallowed RPC access, Not listed in dfs.cluster.admintrators"

Products

• Services Suite

Issue/Introduction

Symptoms:

Namenode failover commands fails with "Disallowed RPC access"

[gpadmin@hdm1 ~]$ hdfs haadmin -failover nn2 nn1
Operation failed: Disallowed RPC access from [email protected] (auth:KERBEROS) at 3.14.144.68. Not listed in dfs.cluster.admintrators
 at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.checkRpcAdminAccess(DFSZKFailoverController.java:190)
 at org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:93)
 at org.apache.hadoop.ha.protocolPB.ZKFCProtocolServerSideTranslatorPB.gracefulFailover(ZKFCProtocolServerSideTranslatorPB.jav61)
 at org.apache.hadoop.ha.proto.ZKFCProtocolProtos$ZKFCProtocolService$2.callBlockingMethod(ZKFCProtocolProtos.java:1548)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

Environment

Cause

By default, gpadmin / or any other user is not listed as dfs.cluster.administrators which comes into effect after securing the cluster. Only user id listed under dfs.cluster.administrators are allowed to run failover commands.

Resolution

There are 2 ways to get around it depending on your requirements

1. Add gpadmin to the list of administrators if you administer the cluster using gpadmin.

<property>
<name>dfs.cluster.administrators</name>
<value>gpadmin</value>
</property>

2. Use hdfs user to perform the failover if you do not want to add gpadmin to the list of administrator.

sudo -u hdfs kinit -kt <hdfs.service.keytab> <hdfs/hostname@REALM>
sudo -u hdfs haadmin -failover nn2 nn1