Unable to Add New Host to Cluster in SDDC Manager Due to “Failed to Get Network Info” Error

Products

VMware SDDC Manager

Issue/Introduction

When trying to add a host to a cluster in SDDC Manager the following error will be observed at the 'Switch Configuration' step: "Failed to get network info. Http failure response for <vCenter server>/ui/api/v1/clusters/<Cluster-ID>/network/queries/: 500 Internal Server Error"

The domainmanager.log reports an error about a connection failure to the vCenter Server..

/var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDT9:56.914+0000 DEBUG [vcf_dm,1eXXXXXXX,fc53]
[c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7200-exec-3]
Fetching Cluster with ID 791f2aed-f769-4379-ab13-786bfc5ea2eb from inventory

YYYY-MM-DDT9:56.964+0000 INFO [vcf_dm,1eXXXXXXX,fc53]
[c.v.e.s.c.s.ClusterNetworkConfigQueryHelper,http-nio-127.0.0.1-7200-exec-3]
Checking for clusterNetworkConfigurationFuture is completed

YYYY-MM-DDT9:56.964+0000 ERROR [vcf_dm,1eXXXXXXX,fc53]
[c.v.e.s.c.s.ClusterNetworkConfigQueryHelper,http-nio-127.0.0.1-7200-exec-3]
Error while retrieving response object java.util.concurrent.ExecutionException:com.vmware.vcf.common.util.VsphereRuntimeException: Cannot connect to vCenter server.

at java.base/java. util.concurrent. FutureTask. report (FutureTask. java: 122)

at java.base/java. util. concurrent. FutureTask.get (FutureTask. java:191)

at com. vmware. evo. sddc. common. services. ClusterNetworkConfigQueryHelper.getClusterNetworkConfiguration (ClusterNetworkConfigQueryHelper.java:565)

at jdk.internal. reflect. GeneratedMethodAccessor5618. invoke (Unknown Source)

at com. vmware. vcf. clustermanager. controller. v1. ClusterController. getClusterNetworkConfigurationQueryResponse (ClusterController. java: 2954)

at jdk.internal.reflect.GeneratedMethodAccessor5382.invoke (Unknown Source)

at java.base/jdk. internal. reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl. java: 43)

at java.base/java.lang.reflect.Method. invoke (Method. java: 569)

The operationsmanager.log shows : cannot login due to incorrect credential of vCenter service account.

/var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDT2:39.214+0000 ERROR [vcf_dm,6X1234XXXXXXXXX,8e58]
[c.v.e.s.c.c.v.Vsphere.VcManagerBase,dm-exec-16]
Cannot complete login due to incorrect credentials:<vCenter-Service-account-User>

YYYY-MM-DDT2:39.919+0000 DEBUG [vcf_dm,1XXXXXXXXXXXXXXX,128e]
[c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7200-exec-1]
Fetching Cluster with ID <Cluster-ID> from inventory

YYYY-MM-DDT2:39.970+0000 INFO [vcf_dm,1XXXXXXXXXXXXXXX,128e]
[c.v.e.s.c.s.ClusterNetworkConfigQueryHelper,http-nio-127.0.0.1-7200-exec-1]
Checking for clusterNetworkConfigurationFuture is completed

YYYY-MM-DDT2:39.970+0000 ERROR [vcf_dm,1XXXXXXXXXXXXXXX,128e]
[c.v.e.s.c.s.ClusterNetworkConfigQueryHelper,http-nio-127.0.0.1-7200-exec-1]
Error while retrieving response object

java. util. concurrent. ExecutionException: com. vmware. vcf. common. util. VsphereRuntimeException: Cannot connect to vCenter Server

at java.base/java. util. concurrent. FutureTask. report (FutureTask. java: 122)

at java.base/java. util. concurrent. FutureTask. get (FutureTask. java: 191)

at com. vmware. evo. sddc. common. services. ClusterNetworkConfigQueryHelper. getClusterNetworkConfiguration

at jdk. internal. reflect. GeneratedMethodAccessor5618. invoke (Unknown Source)

at java.base/jdk.internal.reflect. DelegatingMethodAccessorImpl. invoke (DelegatingMethodAccessorImpl. java: 43)

at java.base/java.lang.reflect.Method. invoke (Method. java: 569)

Environment

VMware SDDC Manager 5.x

Cause

SDDC Manager cannot connect to vCenter due to the associated service account being disconnected.

Resolution

1. Take snapshot of SDDC manager.

2. Retrieve the associated vCenter service account credential from SDDC manager: ( KB : Retrieve the service accounts credentials from SDDC Manager )

# TOKEN=$(curl -d '{"username" : "<sso_username>", "password" : "<sso_password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

# curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less

3. Login to the target vCenter Server where the service account is defined using the Administrator account and update the password for the associated service account to match what is in the SDDC Manager database.

Edit a vCenter Single Sign-On User

4. Restart SDDC manager services:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

5. Add host to a cluster in SDDC Manager.