Spherelet version compatibility and SHA-1 validation certificate prechecks are failing on the SDDC manager.
search cancel

Spherelet version compatibility and SHA-1 validation certificate prechecks are failing on the SDDC manager.

book

Article ID: 381003

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

Symptoms:

 

  • When running the vCenter upgrade prechecks on the SDDC manager GUI both the Spherelet version compatibility and the SHA-1 validation are failing with the following errors:

    There is an error while performing spherelet version compatibility precheck: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1007)
    High: Do not perform upgrade without addressing this issue.Check log at /var/log/vmware/vcf/operationsmanager/assessment/pythonvalidations/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/artifacts/spherelet-version-mismatch-precheck-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-log.txt.

    vSphere SHA-1 validation failed
    High: Do not perform upgrade without addressing this issue.Check the /var/log/vmware/vcf/operationsmanager/assessment/pythonvalidations/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/artifacts/vsphere-sha1-validation-execution-error-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.txt file for more details. If that file contains error code 'rpc_s_connection_closed' then please retry the precheck as it could not connect to verify whether weak algorithms (e.g. SHA-1) are in use on the vCenter.

  • The vCenter does not report any invalid certificate when running the certificate verification Standalone Precheck Script

  • vSphere for Tanzu is not installed in the vCenter.

  • When we verify the precheck logs spherelet-version-mismatch-precheck.log.txt and vsphere-sha1-validation-execution-error.txt retrieved from the prechecks error messages we are seeing the following error:
     
    ERROR Caught exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1007)
    Traceback (most recent call last):
      File "/opt/vmware/vcf/operationsmanager/scripts/assessment/common-validations/vsphere8_upgrade_certificate_checks.py", line 668, in validate_all_esx_hosts
        si = get_si(cert_store)
      File "/opt/vmware/vcf/operationsmanager/scripts/assessment/common-validations/vsphere8_upgrade_certificate_checks.py", line 344, in get_si
        si = SmartConnect(host=server, port=port_number,
      File "/opt/vmware/vcf/operationsmanager/scripts/assessment/2x-migration/lib/third-party/pyvmomi-7.0.1/pyVim/connect.py", line 842, in SmartConnect
        supportedVersion = __FindSupportedVersion(protocol,
      File "/opt/vmware/vcf/operationsmanager/scripts/assessment/2x-migration/lib/third-party/pyvmomi-7.0.1/pyVim/connect.py", line 720, in __FindSupportedVersion
        serviceVersionDescription = __GetServiceVersionDescription(protocol,
      File "/opt/vmware/vcf/operationsmanager/scripts/assessment/2x-migration/lib/third-party/pyvmomi-7.0.1/pyVim/connect.py", line 641, in __GetServiceVersionDescription
        tree = __GetElementTree(protocol, server, port,
      File "/opt/vmware/vcf/operationsmanager/scripts/assessment/2x-migration/lib/third-party/pyvmomi-7.0.1/pyVim/connect.py", line 608, in __GetElementTree
        conn.request("GET", path)
      File "/usr/lib/python3.10/http/client.py", line 1283, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
        self.send(msg)
      File "/usr/lib/python3.10/http/client.py", line 976, in send
        self.connect()
      File "/usr/lib/python3.10/http/client.py", line 1455, in connect
        self.sock = self._context.wrap_socket(self.sock,
      File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
        return self.sslsocket_class._create(
      File "/usr/lib/python3.10/ssl.py", line 1071, in _create
        self.do_handshake()
      File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1007)

Environment

VMware SDDC Manager 5.1.x
VMware SDDC Manager 4.x

Cause

The prechecks are failing with an exception as SDDC manager is not able to establish a trusted connection to the vCenter to retrieve the VMware Endpoint Certificate Store (VECS) information.

This issue might occur if the vCenter Certificates has been replaced outside of SDDC Manager.

Resolution

Add the CA signing the vCenter Certificate to the SDDC Manager trust store to re-establish the trust between both parties.

Once trust has been established between the SDDC manager and the vCenter reattempt the prechecks.

 

To achieve the task use one of the following procedures:

 

  1. Using the SDDC Manager UI:

    Add a Trusted Certificate to the SDDC Manager Trust Store

    (This functionality is available in VMware Cloud Foundation 4.5.1 and later)


  2. From the SDDC manager command line using one of the following KB articles:

  3. Manual procedure to export the certificate from vCenter and import to SDDC manager:

    (Needs to applied for the full chain of trust of the certificate, Root CA and any sub-CAs)


A. To retrieve the CA from the VECS (in case you don't have the .PEM file available):

        1. Login to the vCenter on SSH

        2. Retrieve the list of certificates from the Trusted Root store using the following command:

          /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

        3. Identify the issuing CA (and any subCA if applicable) from the list and note the aliases.

        4. Export the CA certificates to file:

          /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <CA-Alias> --output /tmp/<CA-name>.crt  

          Note: Replace <CA-Alias> and <CA-name> accordingly with the information retrieved from step 3. Repeat for any SubCA if applicable.

        5. Retrieve the .crt files using a file transfer utility

B. To import in SDDC manager:

        1. Use any file transfer utility to copy root CA certificate file to the /tmp directory on the SDDC-Manager VM.
          SSH to SDDC-Manager and change to the root user:
          su

        2. Run the below command to get commonsvcs truststore password
          cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key

        3. Use the following command to import root CA to commonsvcs truststore:
          keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store
           
          Note: <aliasname> : This can be any user defined name. <certificate file> : Provide the path of root CA Certificate

        4. When prompted, provide the password to add certificate to truststore obtained in step 2.

        5. Type yes when prompted to trust the certificate.

        6. Verify if new root CA is added to common services truststore, enter the password for truststore key obtained in step 2.

          keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store

        7. Use the following command to import root CA to SDDC-Manager truststore:

          keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts 
           
          Note: <aliasname> : This can be any user defined name. <certificate file> : Provide the path of root CA Certificate

        8. When prompted, provide the password to add certificate to truststore.

        9. Type yes when prompted to trust the certificate.

        10. Verify if new root CA is added to SDDC-Manager truststore.
          keytool -list -v -keystore /etc/alternatives/jre/lib/security/cacerts
           
        11. After importing CA certificate to the truststore, refresh the trusted certificates using the API from the SDDC-Manager

          curl -X POST localhost/appliancemanager/trustedCertificates/refresh

Additional Information

In some occasions we might need also to update the SSH host keys on the SDDC Manager to establish the connection:

How to update the SSH host keys on the SDDC Manager

Powered by Wolken Software