Day 2 operations fails for VCF Operations with error LCMCOMMON80063

Products

• VCF Operations

Issue/Introduction

• Day two operations tasks fails with error 

  Verify Ssh Connection
  Request Failed
  LCMCOMMON80063
  Error Code: LCMCOMMON80063
  Invalid root credentials. Provide right credentials and retry
  SSH authorization failed with the provided root password for ###.###.###.###. Make sure the root account and root password aren't expired. Verify the credentials for rootPassword and retry.

• Unable to sign into ssh session as root 
• Password shows as disconnected for VCF Operations node(s) when navigating to Fleet Management → Passwords 
• Running the command chage -l root in single user mode shows that the password has expired 
• Remediating the root password for the VCF Operation appliance succeeds but but still unable to login to ssh session with root password. You can use the following documentation as guide to remediation of password Remediate Passwords for VMware Cloud Foundation Components 

Environment

VCF Operations 9.x

Cause

Root password has expired for one or more nodes 

Resolution

1. Launch the VCF Operations admin UI, for example: https://vcfops-masternode-fqdn/admin/
2. Log in as admin user.
3. Take cluster offline 
4. Take snapshot of all nodes in the cluster including Cloud Proxies
5. Follow step 3 to 17 in kb How to Reset the Root Password for Aria Operations 
6. Sign into ssh session as root and confirm that the password is no longer expired by running the command chage -l root 
7. Remediate the password as per documentation Remediate Passwords for VMware Cloud Foundation Components 
8. Repeat this process for each affected node.