Sync updates in vCenter lifecycle manager fails with error "A general system error occurred: A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot."

Products

• VMware vCenter Server

Issue/Introduction

• Unable to download the latest patches and updates via vCenter lifecycle manager.
  
  
• Following snippets are observed at /var/log/vmware/vmware-updatemgr/vum-server/vum-server.log
YY-MM-DDTHH:MM:SS.697-06:00 error vmware-vum-server[11165] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.SyncDepotsTask] [SyncDepotsTask 221] Failed to sync depots. Merged depot content is invalid: Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.depotContent.ValidationError<A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot.>
YY-MM-DDTHH:MM:SS.698-06:00 info vmware-vum-server[11165] [Originator@6876 sub=PM.AsyncTask.SyncDepotsTask{16}] [vciTaskBase 1496] SerializeToVimFault fault:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcIntegrity.lifecycle.depotContent.ValidationError",
-->          arg = <unset>,
-->          message = <unset>
-->       }
-->    ],
-->    reason = "vLCM Task failed, see Error Stack for details."
-->    msg = "{
-->     "data": null,
-->     "error_type": "ERROR",
-->     "messages": [
-->         {
-->             "args": [],
-->             "default_message": "A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot.",
-->             "id": "com.vmware.vcIntegrity.lifecycle.depotContent.ValidationError"
  
  
• Following snippets are observed at /var/log/vmware/vmware-updatemgr/vum-server/imageservice.log
YY-MM-DDTHH:MM:SS,694 INFO imageService[139873647408704] [DepotCollection 379] Could not download from depot at https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml, skipping (('https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml', '', '(\'https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml\', \'\', "HTTPSConnectionPool(host=\'dl.broadcom.com\', port=443): Max retries exceeded with url: /<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml (Caused by SSLError(SSLCertVerificationError(1, \'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)\')))")'))
YY-MM-DDTHH:MM:SS,695 ERROR imageService[139873647408704] [DepotMgr 51] Unable to connect to depot(s) https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml, https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml, https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml, https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml: ('https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml', '', 'Could not download from depot at https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml, skipping ((\'https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml\', \'\', \'(\\\'https://dl.broadcom.com/<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml\\\', \\\'\\\', "HTTPSConnectionPool(host=\\\'dl.broadcom.com\\\', port=443): Max retries exceeded with url: /<Download_token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml (Caused by SSLError(SSLCertVerificationError(1, \\\'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)\\\')))")\'))')
YY-MM-DDTHH:MM:SS6,695 ERROR imageService[139873647408704] [DepotMgr 51] Unable to connect to depot(s) 


• Running openssl s_client -connect dl.broadcom.com:443 -CApath /etc/ssl/certs/  returns with following error: 
  Verify return code: 20 (unable to get local issuer certificate)


Environment

VMware vCenter Server 8.x

 

Cause

This issue is caused by  broken symlink within /etc/ssl/cert directory on the vCenter Server.

Resolution

Note : Take a snapshot of the vCenter before performing this step. Take offline snapshots of all the vCenters if they are in linked mode. Refer to : VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

 

Follow the steps below to fix this issue: 

• SSH to the vCenter Server Appliance. 
  
  
• cd into /etc/ssl/certs



• Run then following to command to list certificates as well as corresponding symlink.
  ls | grep -e "0a775a30" -e "1001acf7" -e "626dceaf" -e "a3418fda"

• Expected output is following, where files with extension .pem is the certificate and one with extension.0 is the symlink. 
  
  0a775a30.0
0a775a30.pem
1001acf7.0
1001acf7.pem
626dceaf.0
626dceaf.pem
a3418fda.0
a3418fda.pem
  
  
• If a certificate is missing the symlink, Please create one as using the following command syntax : 
  ln -s /etc/ssl/certs/file_name.0 /etc/ssl/certs/file_name.pem (For example if 0a775a30.pem is missing the symlink, than command will be ln -s /etc/ssl/certs/0a775a30.0 /etc/ssl/certs/0a775a30.pem) 
  
  

Creating the missing symlink should resolve the issue.