Day 2 operations fails for VCF Operations with error LCMCOMMON80063

Products

• VCF Operations

Issue/Introduction

• Day two operations tasks fails with error.

  Verify Ssh Connection
  Request Failed
  LCMCOMMON80063
  Error Code: LCMCOMMON80063
  Invalid root credentials. Provide right credentials and retry
  SSH authorization failed with the provided root password for ###.###.###.###. Make sure the root account and root password aren't expired. Verify the credentials for rootPassword and retry.

• Unable to sign into ssh session as root.
• Password shows as disconnected for VCF Operations node(s) when navigating to Fleet Management → Passwords.
• On the node, running the command "chage -l root" in single user mode shows that the password has expired.
• Remediating the root password for the VCF Operation appliance succeeds but but still unable to login to ssh session with root password. You can use the following documentation as guide to remediation of password Remediate Passwords for VMware Cloud Foundation Components.
• Can also receive the error when SSH fails for proxy nodes.
  

Environment

VCF Operations 9.x

Cause

Root password has expired or forgotten for one or more nodes.

Resolution

1. Launch the VCF Operations admin UI, for example: https://<vcfops-masternode-fqdn>/admin/.
2. Log in as admin user.
3. Take cluster offline.
4. Take snapshot of all nodes in the cluster needing to be updated (VCF Operations, Cloud proxies, etc.).
5. Follow steps below for the specific node type:
   • VCF Operations - Steps 3 to 17 in kb How to Reset the Root Password for Aria Operations.
   • Proxy - How to reset the root password in Cloud proxy.
6. Connect to the affected node via SSH as root user and confirm that the password is no longer expired by running the command "chage -l root". 
7. Remediate the password as per documentation Remediate Passwords for VMware Cloud Foundation Components.
8. Repeat this process for each affected node.