VCF 9.1 Installer fails with "The vCenter Server should not be in ELM Remediation: Remove the vCenter Server from the ELM ring"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Though vCenter Server is not part of ELM, when attempting to import existing vCenter Sever into VCF 9.1, the installer validation fails at the vCenter configuration stage with the following error:
  
  VcManager <VC-FQDN>: The vCenter Server should not be in ELM
Remediation: Remove the vCenter Server from the ELM ring


• Logs on SDDC Manager located at /var/log/vmware/vcf/domainmanager/domainmanager.log shows entries similar to:
  
  {"errorCode":"BROWNFIELD_VSPHERE_CHECK.error","arguments":["vCenter Server is in ELM","VcManager <VC-FQDN>: The vCenter Server should not be in ELM","Remove the vCenter Server from the ELM ring"],
  
  
• Logs on vCenter Server located at /var/log/vmware/vapi/endpoint/endpoint.log shows entries similar to:
  
  YYYY-MM-DDThh:mm:ss | ERROR | vAPI-I/O dispatcher-0 | SessionFacade | 6c3af22b-403b-9c65-ae6d-3b12d41f96bd | Unexpected error occurred while executing the call with session [email protected] (internal id 9291fb21) for method com.vmware.vcenter.topology.nodes.list.
com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): upstream connect error or disconnect/reset before headers. reset reason: connection termination


• Logs on vCenter Server located at /var/log/vmware/trustmanagement/trustmanagement-svcs.log shows entries similar to:
  

  YYYY-MM-DDThh:mm:ss [tomcat-exec-285 [] ERROR com.vmware.vapi.protocol.server.msg.json.JsonServerConnection  opId=] Severe internal error
java.lang.OutOfMemoryError: Java heap space

• Running /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers or /usr/lib/vmware-vmafd/bin/dir-cli nodes list on vCenter Server confirms only one node exists in the SSO domain.
  
  Determining replication agreements and status with the Platform Services Controller (PSC)
  dir-cli Command Reference

Environment

VCF 9.1

Cause

This issue is caused by an OutOfMemoryError condition affecting the vmware-topologysvc service on the vCenter Server. As a result, requests to the com.vmware.vcenter.topology.nodes.list API fails, returning HTTP response with status code 503 errors during the VCF vCenter import workflow, which leads to the import task failure.

Resolution

To resolve this issue, you must ensure the vCenter vAPI services are healthy and responsive before retrying the validation.

1. SSH to the affected vCenter Server as root.
2. Check the status of following services 

   service-control --status vmware-vapi-endpoint
   service-control --status vmware-topologysvc
   service-control --status vmware-vmcam

3. If services are stopped restart those services.
   

   service-control --restart vmware-vapi-endpoint
   service-control --restart vmware-topologysvc
   service-control --restart vmware-vmcam

4. Else, reboot the vCenter Server.
5. The installer will re-query the topology, once it receives a expected response, showing a single node, the ELM error will clear.