Manual registration of vCenter root CA certificates on vCenter Cloud Gateway
search cancel

Manual registration of vCenter root CA certificates on vCenter Cloud Gateway

book

Article ID: 338850

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

The machine SSL certificate on vCenter can be updated on a regular basis. It can be renewed using vCenter generated certificate and signed by VMCA, or be replaced using a custom certificate signed by an external 3rd party CA. When vCenter certificate is updated, VC Discovery Agent will detect the certificate change and try to validate if the new certificate can still be trusted. This validation process is done based on the trusted CA root certificates available on the gateway. If VCDA can trust the new certificate, it will update the vCenter registration accordingly on the gateway and vCenter will continue accessible from the gateway without interruption. The vCenter trusted CA root certificates are fetched and copied to the gateway when the vCenter is registered to the gateway for the first time. So if there is no change in the root CA certificate used in the new machine SSL certificate, which is mostly the case, there will be no interruption and no additional operation need to be done on the gateway.
 

However, in the case GW cannot validate the certificate chain due to an unknown root CA certificate being used (either untrusted 3rd party CA or self-signed) in the new certificate, communication between vCenter and gateway will be broken and vCenter status on gateway marked to "Revalidate the certificate". In this condition, vCenter status can be observed from the https://<GW>:5484/registervc/list below. 


 

To re-establish communication between the gateway and vCenter, the new root CA certificate must be registered to the gateway trust store.

This document describes the manual procedure to obtain root CA certificates from vCenter and register them on the gateway. Please note that the procedure provided here is somehow complex, but updating the root CA certificates will be done through UI in the future release.

 

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 6.7.x

Resolution

Login into the gateway machine via SSH and create a working directory

1. Login as a root user

$ ssh -l root ${GATEWAY_IP}

2. Creating a working directory for saving certificates and temporary files

# mkdir cert-update
# cd cert-update

 

Obtaining trusted root CA certificates from vCenter

In all commands mentioned in this section, please replace ${USERNAME}, ${PASSWORD}, ${VC_HOSTNAME}, ${VC_PORT}, ${CHAIN} variables accordingly.

1. Login and create a session from vCenter

# curl -k -c cookie.dat -X POST -u "${USERNAME}:${PASSWORD}" -H "Accept: application/json" https://${VC_HOSTNAME}:${VC_PORT}/rest/com/vmware/cis/session

2. Get trusted root chains identifiers from vCenter 

# curl -k -b cookie.dat -X GET -H "Accept: application/json" -o output.json https://${VC_HOSTNAME}:${VC_PORT}/api/vcenter/certificate-management/vcenter/trusted-root-chains

Please note that the output.json saved from the command above may contain multiple chains. For example

# cat output.json
[{"chain":"F9ED41028266B68D7DAF63B2A77E880ABFE540E9"},{"chain":"45ACADBC4DC14E59AAF7997B320B731DB06F7D5D"}]


3. Obtain trust root chains from vCenter

For each chain in output.json, execute the following command:

# curl -k -b cookie.dat -X GET -H "Accept: application/json" -o chain-${CHAIN}.json https://${VC_HOSTNAME}:${VC_PORT}/api/vcenter/certificate-management/vcenter/trusted-root-chains/${CHAIN}

Here ${CHAIN} is the value from output.json. For example, using the example ouput.json above, the values are F9ED41028266B68D7DAF63B2A77E880ABFE540E9 and 45ACADBC4DC14E59AAF7997B320B731DB06F7D5D. The command above will create chain-F9ED41028266B68D7DAF63B2A77E880ABFE540E9.json and chain-45ACADBC4DC14E59AAF7997B320B731DB06F7D5D.json file.



Extracting certificate and creating PEM files

The chain-${CHAIN}.json contains root CA certificates and CRL. Here, we need to create a certificate file as a PEM file from the certificate part and ignore the rest. The certificate part starts with "-----BEGIN CERTIFICATE-----" and ends with "-----END CERTIFICATE-----" text. The JSON file also contains "\n" string as a new line, so we must replace this text with actual newline characters.

1. Obtain the certificate part and replace newline characters

This command will get the certificate part and save it to a separate file with a PEM file extension.

# cat chain-${CHAIN}.json | sed -e 's/.*\(-----BEGIN CERTIFICATE-----.*-----END CERTIFICATE-----\\n\).*/\1/g' | sed -e 's/\\n/\n/g' > chain-${CHAIN}.pem


2. Renaming certificate PEM file using certificate's SHA-256 fingerprint

The certificate's SHA-256 fingerprint can be obtained using the following command

# openssl x509 -in chain-${CHAIN}.pem -noout -fingerprint -sha256 | cut -f2 -d= | tr -d ":" | tr "[:upper:]" "[:lower:]"
<sha256-fingerprint>

Then rename chain-${CHAIN}.pem file to <sha256-fingerprint>.pem

# mv chain-${CHAIN}.pem <sha256-fingerprint>.pem

Example:

# openssl x509 -in chain-45ACADBC4DC14E59AAF7997B320B731DB06F7D5D.pem -noout -fingerprint -sha256 | cut -f2 -d= | tr -d ":" | tr "[:upper:]" "[:lower:]"
58d49e299e908cffcc9239b111b020987b6dc1db18f78a7e48e8908e723525b8
 
# mv chain-45ACADBC4DC14E59AAF7997B320B731DB06F7D5D.pem 58d49e299e908cffcc9239b111b020987b6dc1db18f78a7e48e8908e723525b8.pem



Updating ca-bundle.crt file

1. Update certificate PEM file ownership

All certificate files are managed by vc-discovery-agent, which is running as vcda_user user and aap group, so make sure the new files are with the correct owner.

# chown vcda_user:aap <sha256-fingerprint>.pem

2. Copy the new PEM files to the certificate directory

# cp -p <sha256-fingerprint>.pem /var/vmware/aca/certs

3. Append the newly added PEM files contents to ca-bundle.crt

# cat /var/vmware/aca/certs/<sha256-fingerprint>.pem >> /var/vmware/aca/certs/ca-bundle.crt

4. Verify vCenter connection status

After the last step, VCDA will automatically detect the CA bundle file update and revalidate the certificate. Check https://<GW>:5484/registervc/list again and confirm that the communication status returns to "Connected."

 

Additional Information

If VCenter didn't return to "Connected" status after applying the procedure above, there are some points that need to check:

  1. Ensure that the number of PEM files copied to /var/vmware/aca/certs matches to the number of certificate IDs returned in Step 2.2.2.
  2. Check if all PEM files are well-formatted. It should start with "-----BEGIN CERTIFICATE-----" and with "-----END CERTIFICATE-----" and each line is no longer than 64 characters long. There must be a newline character at the end of the file.

  3. If /var/vmware/aca/certs/ca-bundle.crt contents are destroyed due to accidental operation, you can recreate the file using the following procedure:

    # cd /var/vmware/aca/certs
    # cat /etc/pki/tls/cert/ca-bundle.crt *.pem > ca-bundle.crt
  4. Check if vCenter is registered using the IP address instead of the hostname (FQDN).
    If it's the case, ensure the new certificate contains Subject Alternative Name (SAN) with this IP address. VCDA enables the hostname verification during certificate validation. If the certificate doesn't contain the correct IP address, VCDA will not able to trust the new certificate from vCenter.


Powered by Wolken Software