icm deployment hung with no error in GPHDNodeInstaller log

Products

• Services Suite

Issue/Introduction

Symptoms:

When you do icm_client dpeloy or icm_client uninstall to install or uninstall a PHD cluster, you may see one or more cluster nodes hung and icm_client timed out eventually. From /tmp/GPHDNodeInstaller.log on a problematic cluster node, you may find the following lines:

[INFO] Cleaning up old status files [INFO] Checking yum repo file /etc/puppet/puppet.conf.icmbackup.1392926025 [INFO] Cleaning up old status files [INFO] Checking yum repo file /etc/puppet/puppet.conf.icmbackup.1392926027 [INFO] Cleaning up old status files [INFO] Checking yum repo file /etc/puppet/puppet.conf.icmbackup.1392926029
[INFO] Cleaning up old status files
[INFO] Checking yum repo file
/etc/puppet/puppet.conf.icmbackup.1392926025
[INFO] Cleaning up old status files
[INFO] Checking yum repo file
/etc/puppet/puppet.conf.icmbackup.1392926027
[INFO] Cleaning up old status files
[INFO] Checking yum repo file
/etc/puppet/puppet.conf.icmbackup.1392926029

There were no other type of log messages or error in this log.

There were no errors indicate this problem in /var/log/gphd/gphdmgrgphdmgr-webservices.log on admin node.
icm_client scanhosts command showed all node are good.
Multiple files like puppet.conf.icmbackup.1392926025 were created in /etc/puppet.

Verify
When running "puppet cert list" from failed cluster node you will see this error:

Got 2 failure(s) while initializing: change from absent to directory failed: Could not set 'directory on ensure: Could not find group puppet; Could not evaluate: Could not find group puppet

Environment

Cause

User account and group puppet doesn't exist on the cluster node.

Resolution

To fix manually use command useradd and groupadd to add the puppet user and group on the problematic cluster node. It's recommended that you grep the puppet from /etc/passwd and /etc/group on a good node, then use the same info such as home dir etc on the /etc/passwd and /etc/group on the problematic node. Keep the UID and GID that was created by useradd and groupadd commands in those files.