Namenode startup fails with "Does not contain a valid host:port authority: ${secondarynamenode}:50090" on HA cluster after kerberos enablement

Products

• Services Suite

Issue/Introduction

Symptoms:

Namenode startup fails with Does not contain a valid host:port authority: ${secondarynamenode}:50090 on PHD HA (High Availability) cluster after kerberos enablement

2014-01-26 20:15:55,455 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://phd11-dn-1.saturn.local:8480/getJournal?jid=phd11krbha&segmentTxId=1&storageInfo=-40%3A848004040%3A0%3ACID-f9374af6-70d7-4498-bdd1-e407ea48939f failed with status code 500
 Response message:
 getedit failed. java.lang.IllegalArgumentException: Does not contain a valid host:port authority: ${secondarynamenode}:50090 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:193) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:160) at o

Environment

Resolution

Once Kerberos is enabled on a PHD HA cluster, you may see the error reported above when namenode starts. Although, prior to kerberization, the cluster may be be working fine.

In order to workaround the issue, remove the below entries for secondary namenode parameter from /etc/gphd/hadoop/conf/hdfs-site.xml file on all the nodes and restart the services.

<property>
 <name>dfs.namenode.secondary.http-address</name>
 <value>${secondarynamenode}:50090</value>
 <description>
 The secondary namenode http server address and port.
 </description>
</property> 

This parameter is a part of Install & Configuration Manager (ICM) cluster configuration hdfs-site.xml template. If HA cluster has been deployed after removing it from the template, then you will not come across this error.