Lifecycle Manager Unable to Sync Updates After Integrating Authenticated Token When Proxy is In Use

Products

• VMware vCenter Server

Issue/Introduction

When using an HTTPS proxy server, after integrating the VCF authenticated token, you see that Lifecycle Manager is unable to sync updates. 

 Symptoms:

• "Sync Updates" jobs fail in Lifecycle Manager. 
• On the vCenter, in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-<##>.log you see SSL verification failures against dl.broadcom.com.
• When SSH'd into vCenter, you run the command curl -kvvv https://dl.broadcom.com and you are returned with the error "SSL certificate verify result: unable to get local issuer certificate."

Environment

VMware vCenter Server 7.0.x 

VMware vCenter Server 8.0.x 

Cause

This issue occurs when vCenter Server is unable to verify if the proxy certificate was signed by a trusted certificate authority (CA).

Usually this is due to none of certificates in the certificate chain for the proxy server having been published to the vCenter's trusted root certificate store.

Resolution

METHOD 1: Create a rule on your proxy server to bypass SLL inspection for traffic between the vCenter Server and the download repository URL dl.broadcom.com.

METHOD 2: Upload the the CA certificates in the certificate chain for the proxy to your vCenter Server's Trusted Root store

1. Upload Root and Intermediate proxy certs to to /tmp on the vCenter Server Appliance. The file format needs to be x509-encoded base64.
2. Run these commands to add the Root and Intermediate certs (if applicable) for the proxy to the VC, changing out the highlighted variable values with the file names of the certs:

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/<ROOT_CA_CERTIFICATE> --login [email protected] --password <SSO_Admin_Password>
   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/<INTERMEDIARY_CA_CERT> --login [email protected] --password <SSO_Admin_Password>

3. Run this command to refresh the VECS store so the vCenter recognizes the new certs:

   # /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

4. Run a Lifecycle Manager re-sync from the vSphere GUI:
   1. Navigate to Lifecycle Manager Interface 
   2. Click Actions
   3. Click Sync Updates

 

Note: In order for vCenter to accept a certificate as CA certificate and to allow publishing it to the trusted root certificate store, the certificate must fulfill at least 2 requirements:

• The option CA:TRUE must be set
• The X509v3 Key Usage field must include the option "Certificate Sign"

If any of these options is not set, you will not be able to publish the certificate. This can be the case when the proxy service is using a self-signed certificate. These certificates can be marked as CA certificates using the option CA:TRUE, but they won't include the "Certificate Sign" key usage. In order to verify this, you can review the certificate text using the the following command:

# openssl s_client -connect <proxy_FQDN_or_IP>:<proxy_port> | openssl x509 -noout -text

If the output shows the same strings for issuer and subject, the certificate is self-signed. In this case you can either select method 1 instead, or alternatively replace the proxy certificate against a CA-signed certificate and then import the CA root certificates into the trusted root certificate store by following the steps for method 2.

 

To check if the vCenter is downloading, run the command below from the vCenter. You should see entries relating to package downloads originating from dl.broadcom.com:

• tail -f /var/log/vmware/vmware-updatemgr/vum-server/imageservice.log