Error: "Installation failed due to an exception"

Products

• VMware Cloud Director
• VMware Telco Cloud Platform

Issue/Introduction

• vCenter Server Patching fails with "Installation failed due to an exception"
• /var/log/vmware/applmgmt/applmgmt.log shows message "Error: Failed to get machine certs.. Details: unidentifiable C++ exception"
• /var/log/vmware/vmafdd/vmafdd.log shows "Unable to bind to rpc server retry after 1 second"

Environment

VC: 7 Update 3w

TCP: 5

Cause

The patch update fails due to a vmafdd RPC bind failure preventing the appliance management framework and vmware-statsmonitor service from retrieving machine certificates from the VMware Endpoint Certificate Store (VECS), which results in a fatal unhandled C++ exception during the SSO domain query phase of the upgrade.

Resolution

Workaround (To restore services outside of a maintenance window):

1. Exit the appliance shell by running the appliancesh command.

2. Unstage the software packages by executing software-packages unstage.

3. Manually restart all vCenter services by executing: service-control --stop --all && service-control --start --all.
   
   Note: If these steps do not restore the services, rollback the appliance to the latest backup

Resolution (To resolve the root cause and complete the patch update):

1. Log in to the vCenter Server Appliance via SSH.

2. Validate certificate health across all VECS stores by executing the following command: for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After"; done

3. Identify any expired or invalid certificates, specifically focusing on the Machine SSL or STS certificates.

4. If expired certificates are identified, utilize the vSphere Certificate Manager (/usr/lib/vmware-vmca/bin/certificate-manager) to renew or replace them.

5. Restart the affected services and verify that vmafd and vmware-statsmonitor start cleanly without logging further C++ or RPC bind exceptions.

6. Re-initiate the patch sequence from the Appliance Management Interface (VAMI).