SDDC Manager fails to upgrade the NSX Edge node due to an incorrect subject alternative name of NSX Manager node

Products

• VMware SDDC Manager
• VMware NSX

Issue/Introduction

• While upgrading NSX edge node from the SDDC Manager, the upgrade task fails with messsage: "NSX UPGRADE STAGE UPGRADE EDGE - Failed" in the SDDC manager > NSX upgrade page.
• operationsmanager.log in the SDDC Manager (/var/log/vmware/vcf/operationsmanager/operationsmanager.log) reports the SAN name for one or more of the NSX manager node as incorrect.

YYYY-MM-DDTHH:SS DEBUG [vcf_om,68eaa###########################,cdec] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-6] DomainCertificateOperation: {"workflowId":"c4a504a9-####-####-####-############","domainName":"<SDDC_domain_name>","operationType":"GENERATE_CSR","operationStatus":"FAILED","resourceCertificateOperations":[{"resource":{"hostName":"<NSX_Manager_Node_FQDN>","resourceType":"nsxt_manager","master":false},"result":{"status":"FAILED","message":"{\"code\":\"CERTIFICATE_CSR_GEN_FAILED\",\"args\":[\"*****\",\"I/O error on GET request for \\\"https://<NSX_Manager_Node_FQDN>/api/v1/cluster/nodes/deployments\\\": Certificate for <NSX_Manager_Node_FQDN> doesn't match any of the subject alternative names: [<Incorrect_NSX_Manager_Node_SAN>,<NSX_Manager_Node_IP_Address>]\"]}"},"creationTimestamp":176#########,"updateTimestamp":176##########}],"retryOperation":false}

• upgrade-coordinator.log in the problematic NSX Manager node (/var/log/upgrade-coordinator/upgrade-coordinator.log) will have the following snippets.

YYYY-MM-DDTHH:SS ERROR task-executor-13-1-workitem-EDGE-51ac####-####-####-####-############ UpgradeAgentMessagingServiceImpl 3594 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30033" level="ERROR" subcomp="upgrade-coordinator"] Download and verify bundle failed on node 51ac####-####-####-####-############. Failure msg: * Trying (with httplib) <NSX_Manager_Node_FQDN>:443...
* certificate verification 45f3c########################### from <NSX_Manager_Node_FQDN>:443 failed: SSL: no alternative certificate subject name matches target host name '<NSX_Manager_Node_FQDN>'
* Closing connection 0 curl_wrapper: (51) SSL: no alternative certificate subject name matches target host name '<NSX_Manager_Node_FQDN>'

Environment

• VCF 5.x
• NSX 4.x

Cause

• The NSX Edge node upgrade could not be initiated due to an incorrect Subject Alternative Name (SAN) in the NSX Manager node certificate, which prevented a trust relationship from being established between the NSX Manager node and SDDC Manager.

Resolution

Note: Take backup of the NSX Manager nodes before proceeding.

(1) If the NSX Manager nodes are using self-signed certificate:

• • Renew the problematic NSX Manager certificate using Scripted process to replace expired or self-signed VMware NSX Manager Certificates with VMCA-Signed Certificates from the vCenter Server Appliance Shell.
  • Once the certificate is renewed, refresh the SDDC Manager UI and proceed with the upgrade.

(2) If the NSX Manager nodes are using custom certificate:

• • Follow the document to create CSR and import the Custom certificate in to the NSX Manager: Import a Self-signed or CA-signed Certificate
  • Import the NSX Manager' certificate in to the SDDC Manager by following:

1. 1. 1. Take Snapshot of SDDC Manager Appliance without virtual machine memory.
      2. Use a file transfer utility to copy the following certificates to the /tmp directory on the SDDC Manager Appliance.
         1. NSX Manager's full certificate chain.
         2. Another certificate file for only the Root certificate.
      3. SSH to the SDDC Manager Appliance as the vcf user and then issue the su - command to switch to the root user.
      4. Obtain the trusted certificates key by issuing the following command

         KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)

      5. Confirm the output by using the below command:

         echo $KEY

      6. Run the below command to import certificate for the NSX Manager into the SDDC Manager trust store:

         keytool -importcert -alias <aliasname> -file <certificate_file_path> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass $KEY

         • Note:
           Type yes when prompted to trust the certificate.
           The <aliasname> can be any name.
           Replace <certificat_file_path> with the full path to the certificate file that was uploaded in the step 2a.
      7. Run the below command to import the root certificate into the java trust store:

         keytool -importcert -alias <aliasname> -file <root_certificate_file_path> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

         • Note:
           Type yes when prompted to trust the certificate
           The <aliasname> can be any name.
           Replace <root_certificate_file_path> with the full path to the certificate file that was uploaded in the step 2b.
           
           
      8. Restart SDDC services by using following sddcmanager_restart_services script:

         /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

      9. Verify that the new NSX Manager certificate has been added to the SDDC Manager trust store:

         keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY

      10. Proceed to upgrade the NSX Edge nodes.