ICM reconfigure or upgrade fails with error "No entity found for query"

Products

• Services Suite

Issue/Introduction

Symptoms:

Upgrade or reconfigure error

-bash-4.1$ icm_client reconfigure -l applphdprd -c /apps/sysadmin/upgraded_conf
Please make sure you have stopped the cluster. (Press 'y' to continue update, any other key to quit): y
Reconfiguring cluster (This might take a few mins)
Please enter the root password for the cluster nodes:
PCC creates a gpadmin user on the newly added cluster nodes (if any). Please enter a non-empty password to be used for the gpadmin user:
Verifying input

[ERROR] Failed to update the cluster. Reason: Server error: Exception while validating cluster action: No entity found for query

/etc/gphd/gphdmgr/gphmgr-webservices.log

2014-10-13 10:56:59,077 INFO com.greenplum.gphdmgr.icm.cluster.ClusterHelper: obtaining stackname for cluster id 3 and stack type phd
2014-10-13 10:56:59,090 ERROR com.greenplum.gphdmgr.icm.cluster.ClusterHelper: CLUSTER_ACTION_VALIDATION
2014-10-13 10:56:59,090 ERROR com.greenplum.gphdmgr.icm.cluster.ClusterHelper: No entity found for query
2014-10-13 10:56:59,091 ERROR com.greenplum.gphdmgr.icm.cluster.ClusterHelper: [Ljava.lang.StackTraceElement;@623e1ec
2014-10-13 10:56:59,096 ERROR com.greenplum.gphdmgr.icm.web.resource.ICMWebServices: Exception while validating cluster action: No entity found for query

Environment

Cause

This error happens during the ICM validation of the clusterconfig.xml file. ICM will attempt to query the xml data for specific tags and if the xml format is malformed or misconfigured then you will see this type of error

Resolution

There are few way to determine if your xml file is malformed. But here are a couple of use case where we have seen this problem

• User uncommented a param but only removed <!--?" and forgot to remove the ending tag "-->"
• User added PXF in the host hostRoleMapping under a sub tag like hdfs

  This is bad:
  <hostRoleMapping>
  	<hdfs>
  		<namenode>hdm1.phd.local</namenode>
  		<pxf>
  			<pxf-service>hdm1.phd.local</pxf-service>
  		</pxf>
  	</hdfs>
  </hostRoleMapping>
  
  This is good:
  <hostRoleMapping>
  	<hdfs>
  		<namenode>hdm1.phd.local</namenode>
  	</hdfs>
  	<pxf>
  		<pxf-service>hdm1.phd.local</pxf-service>
  	</pxf>
  </hostRoleMapping>
  

Some useful troubleshooting tools that can help with xml file debugging

• xmllint clusterconfig.xml: this cammand will parse an xml input and print out the file in a formatted output. If there are any pasring errors then they will be presented to the user with line number.
• Cat the xml file and try to find the bad "-->" tag

  cat clusterConfig.xml  | egrep -- "-->" | egrep -v -- "<.--" | egrep "<"