VMware Live Recovery Plugin 9.0.4 Intermittently Undeploys with Certificate_Unknown Errors

Products

• VMware Live Recovery

Issue/Introduction

The VMware Live Recovery plugin, specifically version 9.0.4, is observed to intermittently undeploy from the vSphere Client after approximately 30 minutes of operation. This behavior necessitates frequent manual reconfigurations of the plugin to temporarily restore its functionality, only for the issue to recur. Multiple instances of the Live Recovery plugin may also be visible under vCenter Administration -> Client Plugins.

Investigation into the /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log reveals DOWNLOAD_FAILED errors, specifically:
DOWNLOAD_FAILED: Error downloading plugin package com.vmware.drui.polugin:9.0.4 from https://liverecoveryFQDN:443/drplugin/plugin.json.zip
Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown
Caused by: java.security.cert.CertificateException: unable to construct a valid chain
Caused by: java.security.cert.CertPathBuilderException: No issuer certificate for certiciate in certification path found.

Environment

VMware Live Recovery 9.0.4

Cause

The intermittent undeployment and associated errors are primarily caused by a combination of factors:

1. Certificate Trust Issue: The vSphere UI is unable to consistently establish a trusted connection to the Live Recovery FQDN to download or validate the plugin package due to an invalid or untrusted SSL certificate chain, leading to the certificate_unknown and CertPathBuilderException errors.
2. Conflicting Plugin Files: The presence of an older or conflicting version of the Live Recovery plugin (e.g., 9.0.2 alongside 9.0.4) within the vSphere UI's plugin repository creates an unstable environment, contributing to the 9.0.4 plugin's intermittent failure and undeployment. This conflict can lead to confusion during plugin registration and validation processes.

Resolution

To resolve this issue, you need to manually remove the older, conflicting plugin directory and then restart the vSphere UI service.

Steps:

1. Access the vCenter Server: Connect to the vCenter Server Appliance (VCSA) via SSH or console.
2. Navigate to the Plugin Directory: Change directory to where vSphere UI plugin packages are stored:
   cd /etc/vmware/vsphere-ui/cm-service-packages/com.vmware.cis.vsphereclient.plugin

   
   
3. Identify and Remove the Old Plugin:
   • List the contents of the directory to identify any older or extraneous Live Recovery plugin directories. Based on the logs and resolution notes, look for com.vmware.vrUi followed by an older version number.
   • Remove the directory associated with the older, potentially conflicting Live Recovery plugin version (e.g., 9.0.2.24401761).
   • Note: The exact version number in the directory name may vary. Ensure you target the correct older version.
     rm -rf com.vmware.vrUi-9.0.2.24401761
     
     
   • Caution: Double-check the directory name before executing the rm -rf command to avoid deleting critical files.
     
     
4. Restart the vSphere UI Service: Force a clean re-initialization and re-registration of plugins by restarting the vSphere UI service:
   service-control --restart vsphere-ui
   
   
5. Verify: After the service restarts (this may take a few minutes), log back into the vSphere Client and verify that the Live Recovery plugin is deployed and remains stable.

Additional Information

The presence of multiple versions of the same plugin (e.g., 9.0.2 alongside 9.0.4) can cause conflicts within the vSphere UI's plugin management system. By removing the older plugin directory, any stale or conflicting files are eliminated. Restarting the vsphere-ui service then forces the vCenter UI to perform a clean re-discovery and re-registration of its plugins, ensuring that only the intended 9.0.4 Live Recovery plugin is present and loaded without interference from older versions. This process often resolves the underlying certificate trust issues by forcing a clean plugin installation and validation process.