vCenter Server certificate validation error for external solutions in environments with vCenter Server 7.0
search cancel

vCenter Server certificate validation error for external solutions in environments with vCenter Server 7.0

book

Article ID: 322183

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Some solutions, such as VMware vCenter Site Recovery Manager, VMware vSphere Replication, or VMware vCenter Support Assistant are always installed on a different machine than the vCenter Server system.

If you replace the machine SSL certificate of a vCenter Server system with an embedded Platform Services Controller, a connection error results when the solution attempts to connect to the vCenter Server system. The reason is that the vCenter Server system uses a new certificate, but the corresponding registration with the VMware Lookup Service is not updated. When solutions connect to vCenter Server, they use the service registration information, which includes the service URL and the sslTrust string. The sslTrust string is the Base 64 encoded certificate.
 
This article explains how to resolve the issue in environments with vCenter Server 7.0 which is by default with Embedded Platform Services Controller.

Refer to below KBs for 6.x environments:

Environment

VMware vCenter Server 7.0.x

Resolution

Task 0: Validating the sslTrust Anchors for the vCenter Server

Validating the sslTrust Anchors from Command Line on the vCenter Server Appliance
  1. Log in to the vCenter Server Appliance through SSH or console.
  2. Change to Bash shell by typing shell and press Enter.
  3. Run this command to get the current sslTrust anchor stored for the vCenter Server:

    /usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

    For example:

    Note: SSL trust was truncated for readability.

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: 04608398-1493-4482-881b-b35961bf5141
    Site ID: default-site
    Node ID: 59dbed3c-47e3-4038-9b1c-888fff27efd1
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.sso
    Protocol: wsTrust
    URL: https://vcenter.vmware.local/sts/STSService/vsphere.local
    SSL trust: IIDeDCCAmCgAwIBAgIJAP7kGwWSSd0yMA0GCSqGSIb3DQEBCwUAM+++wnWkeAeqnGRYvzifG9M6DK/YMP1S/akAJ...DS7Y85NorISiQfAVy/R2wjP4rNWDfN9DMCcwfPvw/0nFwrpr+0Cg==
  4. Run this command to get the current SSL certificate used on port 443 on the vCenter Server:

    echo | openssl s_client -connect localhost:443

    For example:

    Note: The certificate was truncated for readability.

    CONNECTED(00000003)
    depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
    verify return:1
    depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
    verify return:1
    depth=1 /C=US/DC=vsphere/DC=local/O=vcenter.vmware.local/CN=CA
    verify return:1
    depth=0 /CN=vcenter.vmware.local/C=US
    verify return:1
    ---
    Certificate chain
    0 s:/CN=vcenter.vmware.local/C=US
    i:/C=US/DC=vsphere/DC=local/O=vcenter.vmware.local/CN=CA
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
    ...
    LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
    -----END CERTIFICATE-----
     
  5. Using the output from the openssl s_client and the lstool.py, verify if the returned SSL certificates match for your vCenter Server. If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate to start updating the sslTrust anchors.
Note: If you have more than one vCenter Server in a vSphere domain (Enhanced Linked Mode), repeat the steps for each vCenter Server in the environment.
 

Task 1: Retrieving the Old Certificate

You can retrieve the old certificate using lstool.py (Step 3 of Task 0).
  1. On the vCenter Server, create a directory to store the old certificate. This article uses the location '/certificates'
  2. Run this command to get the old certificate stored for the vCenter Server:

    /usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

    For example:

    Note: SSL trust was truncated for readability.

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: 04608398-1493-4482-881b-b35961bf5141
    Site ID: default-site
    Node ID: 59dbed3c-47e3-4038-9b1c-888fff27efd1
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.sso
    Protocol: wsTrust
    URL: https://vcenter.vmware.local/sts/STSService/vsphere.local
    SSL trust: LIIDeDCCAmCgAwIBAgIJAP7kGwWSSd0yMA0GCSqGSIb3DQEBCwUAM+++wnWkeAeqnGRYvzifG9M6DK/YMP1S/akAJ...DS7Y85NorISiQfAVy/R2wjP4rNWDfN9DMCcwfPvw/0nFwrpr+0Cg==
  3. Copy the Base64 encoded string from field 'SSL trust' to a file and save the file as old_machine.crt
  4. Open old_machine.crt in a text editor.
  5. Append -----BEGIN CERTIFICATE----- to the beginning of the text string, and append -----END CERTIFICATE----- to the end of the text string. Add a carriage return after the 64th character of each line of the contents copied from the sslTrust field.

    For Example:

    -----BEGIN CERTIFICATE-----
    LIIDeDCCAmCgAwIBAgIJAP7kGwWSSd0yMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
    PAMMAkNBMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkW
    QWxvY2FsMQswCQYDVQQGEwJVUzEcMBoGA1UECgwTaG9tZXBzYy5mcml0ei5sb2Nh
    NDAeFw0xNTA4MTAwMDMwMjZaFw0yNTA4MDQwMDMwMjVaMCsxHDAaBgNVBAMME2hv
    HWVwc2MuZnJpdHoubG9jYWwxCzAJBgNVBAYTAlVTMIIBIjANBgkqhkiG9w0BAQEF
    LAOCAQ8AMIIBCgKCAQEAzuf/uVMLwlkUKsMXsUPigqZdrXKzEOEzOQ04q8YgVvDX
    w7MAPSTMZzeUsI6P+/4doZU14zAQTl/6dnbwYg65p9mv7CVJb4QgAJH9xFD+33Ab
    aQX7za/bWPgyxsPtccnn+si8QQDx9mMZbDzF0gjdARvpKWwVv4lln8iZ8wUahyC7
    bxnzc5/oWo4Z3DTruHMnvadHRZWzZTn8YeID06R2g8Yu5c50wXbAvNj3TE4x0Qyv
    fUbABXvv2EdYC5tb3g++L6A6tuWYgl+dr4KJ1G5gLvliECAsWsMwtQXq5nH65JdV
    XvRUVIlajC9OavGkd+ziT3yRibJBu2NJrLQp7ehgmQIDAQABo2IwYDAeBgNVHREE
    FzAVghNob21lcHNjLmZyaXR6LmxvY2FsMB0GA1UdDgQWBBSaRwv8djR7+qg7Wk3A
    zib3C3ArljAfBgNVHSMEGDAWgBRkYn4wsyRye8o14OoE3AOTMus6rzANBgkqhkiG
    9w0BAQsFAAOCAQEAU3X/ZEDXO8yDRJkjrQH0acxoc76QRDv+3s6yCpPFU8HmqU1E
    LmoDq67rHoKZw5ziBR/lGHn5oVHYYuJRFdO/b8NO1t2MnedhAaenqmAr4v0FzH6K
    UCgiLq8+ZMPFBz3qFu2i0I8mG6Yy0ud9T4wWUabgZ1C3sDNkQ+NLHXKVxNrPwgQd
    3KyrNpXgBQ0+ZWY3xvvdW5yOwnWkeAeqnGRYvzifG9M6DK/YMP1S/akAJvXSgEkJ
    PEJ3vlvSRy7l2lvU19upt4O/BAk3ZJ+X5uFtv/4GMdbEVZBCmNDS7Y85NorISiQf
    AVy/R2wjP4rNWDfN9DMCcwfPvw/0nFwrpr+0Cg==
    -----END CERTIFICATE-----
     
  6. Proceed with Task 2 to extract the thumbprint of old certificate /certificates/old_machine.crt.

Task 2: Extracting the Thumbprint from the Old Certificate

You can extract the thumbprint from the command line.
 
Extracting the Thumbprint from the Command Line on the Appliance
  1. Run below command to get the thumbprint:

    openssl x509 -in /certificates/old_machine.crt -noout -sha1 -fingerprint
You see output similar to:

SHA1 Fingerprint=13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88

The thumbprint, is the sequence of numbers and letters that follow the equal sign.
 

Task 3: Retrieving the New Certificate

You can retrieve the new certificate or current machine ssl certificate using vecs-cli:

Retrieving the New Certificate on the vCenter Server Appliance  

  1. Run this command to view the new certificate:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT 
     
  2. Export the certificate to a file with this command:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /certificates/new_machine.crt 

Task 4: Running the ls_update_certs.py Script

Run the ls_update_certs.py script on the vCenter Server. To successfully run the script, you must have both the thumbprint of the old vCenter Server certificate and the new vCenter Server certificate.
 
Warning: Perform a backup or offline snapshot of all the vCenter Servers in Enhanced Linked mode before proceeding with below steps, so that you can recover if problems result.
 

Running ls_update_cert on the Appliance

The ls_update_certs.py script is located at /usr/lib/vmware-lookupsvc/tools/ 

  1. Change directory to /usr/lib/vmware-lookupsvc/tools/ with the following command:

    cd /usr/lib/vmware-lookupsvc/tools/
     
  2. Run this command:

    python ls_update_certs.py --url Lookup_Service_FQDN_of_vCenter_Server --fingerprint Old_Certificate_Fingerprint_from_Task_2 --certfile New_Certificate_Path_from_Task_3 --user [email protected] --password 'Password' 2>/dev/null

    For example (do not copy the fingerprint used in this example):

    python ls_update_certs.py --url https://vcenter.vmware.local/lookupservice/sdk --fingerprint 13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88 --certfile /certificates/new_machine.crt --user [email protected] --password 'Password@123' 2>/dev/null

Sample result:

[email protected] [ /usr/lib/vmware-lookupsvc/tools ]# python ls_update_certs.py --url https://vcenter.vmware.local/lookupservice/sdk --fingerprint 13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88 --certfile /certificates/new_machine.crt --user [email protected] --password 'Password@123' 2>/dev/null
Get service 11c10f43-2adb-4da5-b04c-cf18a66653b6
Update service 11c10f43-2adb-4da5-b04c-cf18a66653b6; spec: /tmp/svcspec_hvu8kxs8
Get service d8011901-5672-4196-ac18-42dc9248c392
Update service d8011901-5672-4196-ac18-42dc9248c392; spec: /tmp/svcspec_kost4p0i
.
.
Get service 5db24673-f515-4578-ad10-a51e9968bfdc
Don't update service 5db24673-f515-4578-ad10-a51e9968bfdc
Get service 933e1223-e4df-43e7-bafe-244b9f4ac918
Don't update service 933e1223-e4df-43e7-bafe-244b9f4ac918
Updated 41 service(s)
[email protected] [ /usr/lib/vmware-lookupsvc/tools ]#


Additional Information

Syntax for ls_update_cert

Run the script using the following syntax:

python ls_update_certs.py --url LS_URL --fingerprint OLD_CERT_SHA1_HASH --certfile NEW_CERT_PEM_FILEPATH --user USER --password PASSWORD
 
LS_URLLookup service URL. On the vCenter Server, use the following URL as a model:
https://external_platform_services_controller_FQDN.example.com/lookupservice/sdk
OLD_CERT_SHA1_HASH
Thumbprint of the certificate that vCenter Server used before certificate replacement acquired in Task 2.
 
Note: VMware does not recommends to find the old vCenter Server certificate in the filesystem.
NEW_CERT_PEM_FILEPATH
PEM encoded file of the new vCenter Server machine SSL certificate acquired in Task 3.
 
Use the file that you just passed in as part of certificate replacement. If you no longer have that file, use the process in Retrieving the New Certificate.
 
Note: Attempting to find the new vCenter Server certificate in the filesystem is not recommended.
USER and PASSWORDUser with administrator privileges for vCenter Single Sign-On.


Powered by Wolken Software