VMware Cloud Foundation HMS or LCM components may fail to backup or upgrade

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• LCM fails to upgrade HMS or LCM agent or components
• The ownership of the /opt/vrack/hms or /home/vrack/lcm directory and/or files under it on the SDDC Manager Controller virtual machine is something other than vrack:vfabric
• In the /home/vrack/vrm/logs/hms-upgrade.log file, you see messages similar to:

     2018-01-07 23:06:58.906 ERROR [main] [com.vmware.vrack.hms.common.util.FileUtil]       Failed to move directory '/opt/vrack/hms' as '/home/vrack/VMware/vRack/nfs-           mount/lcm/repo/hms/upgrade/cf9157c7-75d1-4d49-8304-aa782e6b3387/hms'.
     java.io.FileNotFoundException: /opt/vrack/hms/hms.sh.orig (Permission denied)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

To resolve this issue, ensure that the /opt/vrack/hms or /home/vrack/lcm/lcm-app/conf folder and files are all owned by the vrack user.

1. SSH as the root user to the SDDC Manager Controller virtual machine.
2. Run this command to check permissions on HMS folder and files:

​ls -lah /opt/vrack/hms

Note: You see output similar to:

drwxr-xr-x 8 vrack vfabric 4.0K Jan 12 21:32 .
drwxr-xr-x 3 vrack vfabric 4.0K Nov 27 14:08 ..
drwxr-xr-x 2 vrack vfabric 4.0K Nov 27 14:08 certmgmt
drwxr-xr-x 2 vrack vfabric 4.0K Nov 27 14:08 config
-rw-r--r-- 1 root  root       5 Jan 12 21:32 hms-agent.pid
-rwxr-xr-x 1 vrack vfabric  314 Nov 27 13:48 hms.service
-rwxr-xr-x 1 vrack vfabric 6.3K Nov 27 13:48 hms.sh
-rwxr-xr-x 1 vrack vfabric 6.3K Nov 27 13:48 hms.orig.sh
drwxr-xr-x 2 vrack vfabric  12K Nov 27 14:08 lib
drwxr-xr-x 2 vrack vfabric 4.0K Jan 12 21:32 logs
drwxr-xr-x 2 vrack vfabric 4.0K Nov 27 14:08 plugin-jars
drwxr-xr-x 2 vrack vfabric 4.0K Nov 27 14:08 upgrade-config

 

3. Change to the  /opt/vrack/hms directory:cd /opt/vrack/hms
 

4. Run the following command to change ownership and permissions on the HMS folder and files except the hms-agent.pid file:

chown -R vrack:vfabric `ls | grep -v .pid`
chmod -R 755 `ls | grep -v .pid`

 

5.Run this command to check permissions on LCM configuration folder and files:

ls -lah /home/vrack/lcm/lcm-app/conf
 

​Note: You see output similar to:

drwx------ 2 vrack vfabric 4.0K Mar 16 20:50 .
drwx------ 5 vrack vfabric 4.0K Jan  7 20:46 ..
-rw------- 1 vrack vfabric 1.1K Jan  6 18:45 VersionAlias.yml
-rw------- 1 vrack vfabric 5.8K Feb 11 00:46 application-evo.properties
-rw------- 1 vrack vfabric  405 Jan  6 18:45 cassandra-evo.properties
-rw------- 1 vrack vfabric   12 Jan  6 18:45 keystore.cred
-rw------- 1 vrack vfabric 1.4K Jan  6 18:45 lcm_bundle_trusted_sig.cert
-rw------- 1 vrack vfabric 2.6K Jan  6 18:45 lcm_default_keystore.p12
-rw------- 1 vrack vfabric  397 Jan  6 18:45 lcm_known_hosts
-rw------- 1 vrack vfabric 3.5K Jan  6 18:45 logback-evo.xml
-rw------- 1 vrack vfabric  134 Jan  6 18:45 logback.xml
-rw------- 1 vrack vfabric   70 Jan  6 18:45 zk-evo.properties

   
6. Change to the  /home/vrack/lcm/lcm-app/conf directory:

cd /home/vrack/lcm/lcm-app/conf
 

 7.Run the following commands to change ownership and permissions on the LCM configuration folder and files:

chown vrack:vfabric *
chmod 600 *

8.Run the following commands to restart HMS and LCM services: 
systemctl restart hms
systemctl restart lcm

9.Wait approximately 15 mins before reattempting the upgrade and then reattempt the upgrade in the UI.