Repointing vCenter to a new Single Sign-On domain fails with "Name or service not known"

Products

• VMware vCenter Server

Issue/Introduction

When attempting to re-point vCenter server to a different Single Sign-On domain , the following symptoms are present:

• In /var/log/vmware/cloudvm/cmsso_util.log, we see the following errors logged:

YYYY-MM-DDTHH:MM:SS INFO cmsso_util Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-vpostgres...
Successfully started service vmware-vpostgres
Performing start operation on service vpxd-svcs...
Successfully started service vpxd-svcs

...

YYYY-MM-DDTHH:MM:SS INFO cmsso_util Installing firstboot /usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py
YYYY-MM-DDTHH:MM:SS INFO cmsso_util RC = 1
Stderr = YYYY-MM-DDTHH:MM:SS [Errno -2] Name or service not known

• No core dump present in /storage/core/ after the command failed
• When running cmsso-util domain-repoint with --mode pre-check , no errors are returned
• Name resolution (forward and reverse) succeeds for the vCenter Server in question
• The vCenter Server does not have custom/conflicting DNS records for its FQDN within /etc/hosts

Environment

VMware vCenter Server 6.7.x

Cause

This error occurs when the Certificate Manager attempts to establish a connection to the Authorization (com.vmware.cis.authorization.server) endpoint within vCenter but fails. The cause may be a mismatch between the endpoint configuration or the actual URL, leading to an unsuccessful connection.

Resolution

1. To identify and verify the service endpoints registered with your vCenter Server, please provide the following information::

• • Log bundle: If available, please share the contents of the /commands/lstool.txt file from your log bundle.
  • Live system: Alternatively, please follow the instructions outlined in the knowledge base article: Process to view the List of Services Registered with Single Sign-On (2043509)
    
    
• Once we have access to the registered service endpoints, we can compare the authorization endpoint's FQDN against the vCenter Server's current configuration. This will help us determine if any mismatches exist and address any potential issues related to incorrect FQDNs.
  For example, the below output shows "incorrect-FQDN.incorrectdomain.local" when it should be the same FQDN that vCenter currently is configured for.
  
  Example:
  
  Type: com.vmware.cis.authorization.server
  Protocol: vmomi
  URL: https://incorrect-FQDN.incorrectdomain.local:443/invsvc/vmomi/sdk
  
  SSL Trust:
  
  Note: Other service endpoints should have the correct URL. This KB is specifically targeting scenarios when the authorization endpoint is incorrect.
  
  
• It's important to note that a close resemblance between the vCenter Server FQDN and the Single Sign-On (SSO) domain can lead to mismatches in service endpoint configurations. While this configuration is not recommended due to potential authentication conflicts, it can be addressed by carefully reviewing and aligning the FQDNs. For a comprehensive understanding of vSphere domains, domain names, and sites, please refer to Understanding vSphere Domains, Domain Names, and Sites
   

2. Having confirmed a discrepancy between the expected and actual service endpoint configuration, we will utilize the lsdoctor tool to rectify the Single Sign-On (SSO) endpoints. To ensure a successful repair process, please refer to Using the 'lsdoctor' Tool (80469)

• • To perform the most extensive repair of the SSO endpoints, we will execute the lsdoctor tool with the -r or --rebuild switch. This action will significantly modify the configuration. Therefore, it's crucial to ensure a safe rollback plan is in place before proceeding. (see Impact/Risks section).
    
    
• In addition, another method to re-create the endpoints would be to perform the following:

1. Reconfigure the vCenter Server Primary Network Identifier (PNID) to a different FQDN that does not match the original, or the incorrect FQDN found in the endpoint URL collected above.
2. Once successfully renamed, attempt to re-point (cmsso-util) vCenter Server to the target SSO domain.
3. Once re-pointed, perform Step 1 to rename the vCenter Server PNID to the preferred FQDN. This should automatically correct the endpoints.

Note: When changing the PNID of vCenter, it is advised that a new DNS record must be updated to reflect the change. In addition, ensure forward/reverse lookup resolve successfully.

Additional Information

The following instructions involve modifying the currently registered Single Sign-On (SSO) endpoints.
It is strongly recommended to create backups of your system prior to performing these steps. This will allow for a smooth rollback in case of any unexpected issues or the need to revert changes.