Notes:
- Installing vCenter Server 6.0 update 1b on a system that is affected does not resolve the issue until you replace the certificates again.
- The update resolves the issue for certificate replacement with the Certificate Manager utility. The update does not resolve the issue for certificate replacement from the Services Controller UI.
To work around this issue for environments with an external Platform Services Controller, you must update the lookup service registration for both the vCenter Server system and the Platform Services Controller each time you replace a certificate. You update the service registration with the ls_update_certs.py script.
Notes:
- Run this script always on the Platform Services Controller.
- To run the script, you need the thumbprint of the old vCenter Server certificate and you need the new certificate. You must upload these certificates to the Platform Services Controller before you run the script.
- Be sure to back up your existing certificates before you run the script.
- Run this script each time you replace a certificate.
Solution Overview
How you run the script depends on where you replaced the machine SSL certificate.
Certificate replaced on Platform Services Controller only
|
Run the script on the Platform Services Controller, passing in the old and the new certificate for the Platform Services Controller.
|
Certificate replaced on vCenter Server system only
|
Run the script on the Platform Services Controller, passing in thumbprint of the old vCenter Server certificate and the new vCenter Server certificate
|
Certificates replaced on both vCenter Server system and Platform Services Controller
|
Run the script on the Platform Services Controller twice. Ideally, you run the script once after you replace the Platform Services Controller certificate, and again after you replace the vCenter Server certificate.
|
Task 0: Validating the sslTrust Anchors for the PSC and vCenter
Validating the sslTrust Anchors from Command Line on the PSC Appliance
- Log in to the External Platform Services Controller Appliance via SSH.
- Run this command to enable access the Bash shell:
shell.set --enabled true
- Type shell and press Enter.
If you have replaced the SSL certificates on their Platform Services Controllers:
- Run this command to get the current sslTrust anchor stored for the Platform Services Controller:
/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
For example:
Note: The SSL trust was truncated for readability.
Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ########-####-####-####-########5141
Site ID: vmware
Owner ID: [email protected]
Version: 2.0
Endpoints:
Type: com.vmware.cis.cs.identity.sso
Protocol: wsTrust
URL: https://homepsc.example.local/sts/STSService/vsphere.local
SSL trust: MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV ... LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
- Run this command to get the current SSL certificate used on port 443 on the Platform Services Controller:
echo | openssl s_client -connect localhost:443
For example:
Note: The actual string was shortened significantly to improve readability.
CONNECTED(00000003)
depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
verify return:1
depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
verify return:1
depth=1 /C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
verify return:1
depth=0 /CN=psc.example.local/C=US
verify return:1
---
Certificate chain
0 s:/CN=psc.example.local/C=US
i:/C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
...
LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
-----END CERTIFICATE-----
- If you have more than one vCenter Server in a vSphere domain, repeat the command using the FQDN for any remaining VC nodes.
echo | openssl s_client -connect vcenter2.example.local:443
- Using the output from the above openssl s_client and the lstoolutil.py, verify if the returned SSL certificates match for your vCenter Server(s). If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB) to start updating the sslTrust anchors.
If you have replaced the SSL certificates on their vCenter Servers:
- Run this command to get the current sslTrust anchor stored for the vCenter Servers:
/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.vim 2>/dev/null
For example:
Note: The SSL trust was truncated for readability.
Name: AboutInfo.vpx.name
Description: AboutInfo.vpx.name
Service Product: com.vmware.cis
Service Type: vcenterserver
Service ID: ########-####-####-####-########13a7
Site ID: site
Node ID: ########-####-####-####-########12c2
Owner ID: vpxd-########-####-####-####-########[email protected]
Version: 6.0
Endpoints:
Type: com.vmware.vim
Protocol: vmomi
URL: https://vcenter.example.local:443/sdk
SSL trust: MIIDfjCCAmagAwIBAgIJAMLCByASkdjPMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV...v2zUN54IrTl1lfAA8eb6Xy9miKsYZoYan5mgzb+GGD2yLw==
- Run this command to get the current SSL certificate used on port 443 on the vCenter Server:
echo | openssl s_client -connect vcenter_server_FQDN:443
Use this example as a model:
Note: The actual string was shortened significantly to improve readability.
CONNECTED(00000003)
depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
verify return:1
depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
verify return:1
depth=1 /C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
verify return:1
depth=0 /CN=vcenter.example.local/C=US
verify return:1
---
Certificate chain
0 s:/CN=vcenter.example.local/C=US
i:/C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
1 s:/C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
i:/DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
2 s:/DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
i:/DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
---
Server certificate
-----BEGIN CERTIFICATE----
MIIDfjCCAmagAwIBAgIJAMLCByASkdjPMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
...
v2zUN54IrTl1lfAA8eb6Xy9miKsYZoYan5mgzb+GGD2yLw==
-----END CERTIFICATE-----
- If you have more than one vCenter Server in a vSphere domain, repeat the command using the FQDN for any remaining VC nodes.
echo | openssl s_client -connect vcenter2.example.local:443
- Using the output from the above openssl s_client and the lstoolutil.py, verify if the outputted SSL certificates match for your vCenter Server(s). If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB) to start updating the sslTrust anchors.
Validating the sslTrust Anchors from the Command Line on a Windows PSC Installation
- Connect to the External Platform Services Controller using a remote desktop connection and administrator credentials.
- Click Start > run, type cmd and press OK.
If you have replaced the SSL certificates on their Platform Services Controllers:
- Run this command to get the current sslTrust anchor stored for the Platform Services Controller:
"%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2> NULL
For example:
Note: The SSL trust was truncated for readability.
Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ########-####-####-####-########5141
Site ID: vmware
Owner ID: [email protected]
Version: 2.0
Endpoints:
Type: com.vmware.cis.cs.identity.sso
Protocol: wsTrust
URL: https://psc.example.local/sts/STSService/vsphere.local
SSL trust: MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV ... LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
- Run this command to get the current SSL certificate used on port 443 on the Platform Services Controller:
"%VMWARE_OPENSSL_BIN%" s_client -connect localhost:443
For example:
CONNECTED(00000003)
depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
verify return:1
depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
verify return:1
depth=1 /C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
verify return:1
depth=0 /CN=psc.example.local/C=US
verify return:1
---
Certificate chain
0 s:/CN=psc.example.local/C=US
i:/C=US/DC=vsphere/DC=local/O=psc.fexample.local/CN=CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
...
LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
-----END CERTIFICATE-----
- If you have more than one vCenter Server in a vSphere domain, repeat the command using the FQDN for any remaining vCenter Server nodes.
"%VMWARE_OPENSSL_BIN%" s_client -connect vcenter2.example.local:443
- Using the output from the above openssl s_client and the lstoolutil.py, verify if the outputted SSL certificates match for your Platform Services Controller(s). If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB) to start updating the sslTrust anchors.
If you have replaced the SSL certificates on their vCenter Servers:
- Run this command to get the current sslTrust anchor stored for the vCenter Servers:
"%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.vim 2> NULL
For example:
Note: The SSL trust was truncated for readability.
Name: AboutInfo.vpx.name
Description: AboutInfo.vpx.name
Service Product: com.vmware.cis
Service Type: vcenterserver
Service ID: ########-####-####-####-########13a7
Site ID: vmware
Node ID: ########-####-####-####-########12c2
Owner ID: vpxd-########-####-####-####-########[email protected]
Version: 6.0
Endpoints:
Type: com.vmware.vim
Protocol: vmomi
URL: https://vcenter.example.local:443/sdk
SSL trust: MIIDfjCCAmagAwIBAgIJAMLCByASkdjPMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV...v2zUN54IrTl1lfAA8eb6Xy9miKsYZoYan5mgzb+GGD2yLw==
- Execute the command to get the current SSL certificate used on port 443 on vCenter Server:
"%VMWARE_OPENSSL_BIN%" s_client -connect vcenter_server_FQDN:443
Use this example as a model:
Note: The actual string was shortened significantly to improve readability.
CONNECTED(00000003)
depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
verify return:1
depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
verify return:1
depth=1 /C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
verify return:1
depth=0 /CN=vcenter.example.local/C=US
verify return:1
---
Certificate chain
0 s:/CN=vcenter.example.local/C=US
i:/C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
1 s:/C=US/DC=vsphere/DC=local/O=psc.example.local/CN=CA
i:/DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
2 s:/DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
i:/DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
---
Server certificate
-----BEGIN CERTIFICATE----
MIIDfjCCAmagAwIBAgIJAMLCByASkdjPMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
...
v2zUN54IrTl1lfAA8eb6Xy9miKsYZoYan5mgzb+GGD2yLw==
-----END CERTIFICATE-----
- If you have more than one PSC in a vSphere domain, repeat the command using the FQDN for any remaining PSC nodes.
"%VMWARE_OPENSSL_BIN%" s_client -connect vcenter2.example.local:443
- Using the output from the above openssl s_client and the lstoolutil.py, verify if the outputted SSL certificates match for your vCenter Server(s). If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB) to start updating the sslTrust anchors.
Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB)
You can retrieve the old certificate for the vCenter Server system by connecting to the Platform Service Controller using the Managed Object Browser. You must find the sslTrust field of the ArrayOfLookupServiceRegistrationInfo managed object by performing this procedure:
- Create a directory to store the old certificates on the Platform Services Controller. For the guidance in this article, we will be using the following locations:
Platform Services Controller Appliance |
/certificates/ |
Platform Services Controller on Windows |
C:\certificates\ |
- To open the MOB, go to https://psc.example.com/lookupservice/mob?moid=ServiceRegistration&method=List in a browser.
- Log in with the [email protected] username and password when prompted for credentials.
If you are using a custom name for your vCenter Single Sign-On domain, use that user name and password.
- In the filterCriteria text field, modify the value field to show only the tags <filterCriteria></filterCriteria> and click Invoke Method.
This displays the ArrayOfLookupServiceRegistrationInfo objects.
- Search for the following depending on which certificates are replaced:
vCenter Server |
Search (Ctrl+F) for vc_hostname_or_IP.example.com on the page |
Platform Services Controller |
Search (Ctrl+F) for psc_hostname_or_IP.example.com on the page |
- Find the value of the corresponding sslTrust field. The content of that field is the Base64 encoded string of the old certificate.
Use the following examples as models when updating your Platform Services Controller or vCenter Server trust anchors.
Note: The actual string was shortened significantly to improve legibility
For vCenter Server:
sslTrust |
ArrayofString |
MIIDfjCCAmag...
|
url |
anyURI |
https://vcenter.example.local:443/sdk |
For Platform Services Controller:
sslTrust |
ArrayofString |
MIIDfjCCAbad...
|
url |
anyURI |
https://psc.example.local/sts/STSService/vsphere.local |
- Copy the content of the sslTrust field into a text document. Save this document as old_machine.txt.
- Open the old_machine.txt in a text editor.
- Append -----BEGIN CERTIFICATE----- to the beginning of the text string, and append -----END CERTIFICATE----- to the end of the text string. Add a carriage return after the 64th character of the contents copied from the sslTrust field.
For Example:
-----BEGIN CERTIFICATE-----
LIIDeDCCAmCgAwIBAgIJAP7kGwWSSd0yMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
PAMMAkNBMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkW
QWxvY2FsMQswCQYDVQQGEwJVUzEcMBoGA1UECgwTaG9tZXBzYy5mcml0ei5sb2Nh
NDAeFw0xNTA4MTAwMDMwMjZaFw0yNTA4MDQwMDMwMjVaMCsxHDAaBgNVBAMME2hv
HWVwc2MuZnJpdHoubG9jYWwxCzAJBgNVBAYTAlVTMIIBIjANBgkqhkiG9w0BAQEF
LAOCAQ8AMIIBCgKCAQEAzuf/uVMLwlkUKsMXsUPigqZdrXKzEOEzOQ04q8YgVvDX
w7MAPSTMZzeUsI6P+/4doZU14zAQTl/6dnbwYg65p9mv7CVJb4QgAJH9xFD+33Ab
aQX7za/bWPgyxsPtccnn+si8QQDx9mMZbDzF0gjdARvpKWwVv4lln8iZ8wUahyC7
bxnzc5/oWo4Z3DTruHMnvadHRZWzZTn8YeID06R2g8Yu5c50wXbAvNj3TE4x0Qyv
fUbABXvv2EdYC5tb3g++L6A6tuWYgl+dr4KJ1G5gLvliECAsWsMwtQXq5nH65JdV
XvRUVIlajC9OavGkd+ziT3yRibJBu2NJrLQp7ehgmQIDAQABo2IwYDAeBgNVHREE
FzAVghNob21lcHNjLmZyaXR6LmxvY2FsMB0GA1UdDgQWBBSaRwv8djR7+qg7Wk3A
zib3C3ArljAfBgNVHSMEGDAWgBRkYn4wsyRye8o14OoE3AOTMus6rzANBgkqhkiG
9w0BAQsFAAOCAQEAU3X/ZEDXO8yDRJkjrQH0acxoc76QRDv+3s6yCpPFU8HmqU1E
LmoDq67rHoKZw5ziBR/lGHn5oVHYYuJRFdO/b8NO1t2MnedhAaenqmAr4v0FzH6K
UCgiLq8+ZMPFBz3qFu2i0I8mG6Yy0ud9T4wWUabgZ1C3sDNkQ+NLHXKVxNrPwgQd
3KyrNpXgBQ0+ZWY3xvvdW5yOwnWkeAeqnGRYvzifG9M6DK/YMP1S/akAJvXSgEkJ
PEJ3vlvSRy7l2lvU19upt4O/BAk3ZJ+X5uFtv/4GMdbEVZBCmNDS7Y85NorISiQf
AVy/R2wjP4rNWDfN9DMCcwfPvw/0nFwrpr+0Cg==
-----END CERTIFICATE-----
- Save old_machine.txt as old_machine.crt.
- Move or upload (via WinSCP or another SCP client) the certificate to the Platform Services Controller to the location created in Step 1:
Note: SCP is disabled by default, for more information see Error when uploading files to vCenter Server Appliance using WinSCP (2107727).
Platform Services Controller Appliance |
/certificates/old_machine.crt |
Platform Services Controller on Windows |
C:\certificates\old_machine.crt |
You can now extract the thumbprint from this certificate. Proceed to Task 2.
Task 2: Extracting the Thumbprint from the Old Certificate
You can extract the thumbprint from the command line or by using a certificate viewer tool. After you extract the certificate, you can upload it to the Platform Services Controller.
Extracting the Thumbprint from the Command Line on the Appliance
- Log in to the External Platform Services Controller Appliance via SSH.
- Run this command to enable access the Bash shell:
shell.set --enabled true
- Enter shell and press Enter.
- Run this command to get the thumbprint:
openssl x509 -in /certificates/old_machine.crt -noout -sha1 -fingerprint
- You see an output similar to:
SHA1 Fingerprint=##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
The thumbprint, is the sequence of numbers and letters that follow the equal sign.
Extracting the Thumbprint from the Command Line on a Windows Installation
- Make a remote desktop connection to the External Platform Services Controller.
- Open an administrative command prompt.
- Run this command to get the thumbprint:
"%VMWARE_OPENSSL_BIN%" x509 -in c:\certificates\old_machine.crt -noout -sha1 -fingerprint
- You see an output similar to:
SHA1 Fingerprint=##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
The thumbprint is the sequence of numbers and letters that follow the equal sign.
Extracting the Thumbprint Using a Certificate Viewer Tool
You can extract the thumbprint by performing these steps:
- Open the file with a certificate viewer tool. In Windows, double-click the file to open it in Windows Certificate Viewer.
- Get the SHA1 Thumbprint string. In Windows Certificate Viewer, select the SHA1 Thumbprint field.
- Copy the thumbprint string into a plain text editor and replace the spaces with colons or remove the spaces from the string.
Note: With some text editors, invisible characters are added at the beginning. Delete the first character of the thumbprint and any associated spaces, then type, not paste, the character.
Proceed to Task 3 to retrieve the new certificate.
Task 3: Retrieving the New Certificate
Depending on the replacement you are performing, you need the new vCenter Server certificate, the new Platform Services Controller certificate, or both. See Solution Overview above.
If you did not archive the new certificate, you can retrieve it using vecs-cli:
Retrieving the NewCertificate from the Appliance
- Log in to the vCenter Server or External Platform Services Controller Appliance through console or SSH session.
- Run this command to enable access the Bash shell:
shell.set --enabled true
- Enter shell and press Enter.
- Run this command to view the new certificate:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT –-text
- Run this command to export the new certificate to a file:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /certificates/new_machine.crt
- If you retrieved a vCenter Server certificate, move or upload it (via WinSCP or another SCP client) to the Platform Services Controller before you run the script.
Retrieving the NewCertificate on a Windows Installation
- Connect with remote desktop to the vCenter Server or External Platform Services Controller using administrative credentials.
- Open an administrative command prompt.
- Run this command to view the new certificate:
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry list --store MACHINE_SSL_CERT --text |more
- Run this command to export the new certificate to a file:
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output c:\certificates\new_machine.crt
- If you just retrieved a vCenter Server certificate, move or upload it (via WinSCP or another SCP client) to the Platform Services Controller before you run the script.
Proceed to Task 4 to execute the ls_update_certs script with the information gathers from Tasks 1 through 3.
Task 4: Running the ls_update_certs.py Script
Run the ls_update_certs.py script on the Platform Services Controller. To successfully run the script, you must have both the thumbprint of the old vCenter Server Server or Platform Services Controller certificate and the new vCenter Server or Platform Services Controller certificate.
Note: If the password that you supply includes special characters or spaces, surround it with single quotes (Eg. 'Passw0rd!'). Command will fail with below error message, if it is a problem with the password:
YYYY-MM-DD HH:MM:SS INFO com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor - Provided credentials are not valid.
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
Warning: You cannot undo the actions of this script. Perform a backup or a snapshot of the virtual machine so you can recover if problems result.
Running ls_update cert on the Appliance
The ls_update_certs script is located at /usr/lib/vmidentity/tools/scripts/ls_update_certs.py.
- Log in to the External Platform Services Controller Appliance through console or an SSH session.
- Run this command to enable access the Bash shell:
shell.set --enabled true
- Type shell and press Enter.
- Change directories to /usr/lib/vmidentity/tools/scripts/ with the following command:
cd /usr/lib/vmidentity/tools/scripts/
- Run this command:
python ls_update_certs.py --url Lookup_Service_FQDN_of_Platform_Services_Controller --fingerprint Old_Certificate_Fingerprint_from_Task_2 --certfile New_Certificate_Path_from_Task_3 --user [email protected] --password "Password"
For example (do not copy the fingerprint used in this example):
python ls_update_certs.py --url https://psc.vmware.com/lookupservice/sdk --fingerprint ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:## --certfile /certificates/new_machine.crt --user [email protected] --password "Password"
Running ls_update_cert on a Platform Services Controller Windows Installation
- Make a remote desktop connection to the External Platform Services Controller.
- Open an administrative command prompt