vCenter Server certificate replacement in SDDC Manager fails with "Certificate Installation - Failed" due to broken NSX trust
search cancel

vCenter Server certificate replacement in SDDC Manager fails with "Certificate Installation - Failed" due to broken NSX trust

book

Article ID: 449440

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

vCenter Server certificate replacement workflow fail in VMware Cloud Foundation (VCF) environment integrated with vSphere Lifecycle Manager (vLCM). This failure prevents the automated task from completing in SDDC Manager. 

The following symptoms occur during the certificate rotation process:

  • When replacing the vCenter Server certificate in SDDC Manager by manually downloading the CSR, having it signed by a Certificate Authority, and uploading it, the task fails with "Certificate Installation - FAILED" error
  • Despite the task failure, the vCenter certificate is valid and successfully installed, displaying an "Active" status with valid From and Until dates, as illustrated in the image below. 
  • To verify, navigate to SDDC Workload Domains > Select the specific workload domain > Certificates > Select the vCenter under Resource Type colulmn. 
  • Although the certificate is successfully installed on the vCenter Server, the trust relationship between NSX and vCenter is broken, causing the SDDC Manager task to fail and display Error code 26707: "NSX is integrated with vLCM feature of vSphere system. Cannot disable two way authentication and service account creation functionality." as shown in the image below.
  • The commonsvcs logs show the following entries. It indicates that the Domain Manager service is either unresponsive, failing to start, or crashing, preventing the SDDC Manager UI from updating its task status.

/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

YYYY-MM-DDThh:mm INFO  [common, #####, 0fdf]  [c. v. e. s. i. s. SddcManagerControllerInventoryServiceImpl, http-nio-#.#.#.#-####-exec-#] Get Sddc Controller
YYYY-MM-DDThh:mm INFO  [common, #####, 0fdf]  [c. v. e. s. i. s. VcfServiceInventoryServiceImpl, http-nio-#.#.#.#-####-exec-#] Get all VcfServices
YYYY-MM-DDThh:mm ERROR [common, #####, 0fdf] [c.v. e. s.i. s.VcfServiceInventoryServiceImpl, http-nio-#.#.#.#-####-exec-#] Error while trying to retrieve service http://127.0.0.1/domainmanager/about status, 502 Bad Gateway: 
"<html><EOL><EOL><head><title>502 Bad Gateway</title></head><EOL><EOL><body><EOL><EOL><center><hl>502 Bad Gateway</hl></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"
org. springframework. web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway: "<html><EOL><EOL><head><title>502 Bad Gateway</title></head><EOL><EOL><body><EOL><EOL><center><hl>502 Bad Gateway</hl></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"
            at org. springframework. web. client . HttpServerErrorException. create (HttpServerErrorException. java: 108)
            at org. springframework. web. client. DefaultResponseErrorHandler. handleError (DefaultResponseErrorHandler. java : 186)
            at org. springframework. web. client . DefaultResponseErrorHandler. handleError (DefaultResponseErrorHandler. java: 137)

Environment

VMware SDDC Manager 5.x

Cause

The automated SDDC Manager workflow fails to propagate the new vCenter certificate thumbprint to NSX. In vLCM-enabled clusters, NSX enforces strict security policies that block manual thumbprint updates via the UI. This creates a synchronization deadlock where the UI cannot update the stale thumbprint, and the stale thumbprint prevents cross-component communication.

Resolution

The resolution consists of two parts: Validating the NSX trust restoration and cleaning up the SDDC Manager status.

Restore NSX Trust and Restart SDDC Services

  1. Restore NSX Trust by following the steps outlined in the KB article: Error 26707: Cannot disable two-way authentication when pushing NSX depot on vLCM enabled cluster
    Note: Ensure both Enable Trust and Create Service Account options are enabled
  2. Restart the SDDC services following /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh to restart the full service stack cleanly.

Manual Database Cleanup (If Status Fails to Update)

  1. If the certificate status fails to update after restarting the services, the next step is to perform a clean up of specific entries in the SDDC Manager database.
    1. Take a file-based backup and snapshot of the SDDC Manager Virtual Machine. 
    2. Contact Broadcom Support to perform a manual cleanup of the SDDC Manager database

Additional Information

Review the Certificate Management for VMware Cloud Foundation for more information. 

Powered by Wolken Software