The NSX Edge nodes details show blank under "Edge Clusters" tab in workload domains on SDDC Manger UI

Products

• VMware SDDC Manager

Issue/Introduction

• The NSX Edge nodes details show blank under "Edge Clusters" tab in the workload domains on SDDC Manager UI.

• "Failed to get workload domain NSX edge clusters call information" Error message will appear on Summary Page of each Workload Domain mapped to the Edge Cluster/Nodes:

• However on NSX UI, "Edge Clusters" and "Edge Transport Nodes" will be visible and running healthy.
• You can notice SDDC Manager is able to connect and fetch NSX Edge Cluster nodes details from the logs:
• Using SDDC Manager SSH session check the logs of "vcf-sos.log" under the location "/var/log/vmware/vcf/sddc-support"

YYYY-MM-DDTHH:MM:SS.968+0000 INFO [vcf_sos] [sddcunifiedlogicalinventory.py::get_nsxt_edge_clusters_with_id::####::MainThread] Found NSX edges for ee3fe419-32e1-####-####-03f81f4de44f: [{'cluster_ids': ['fd4b80c8-705d-####-####-97f2c0e881b1'],
 'edge_cluster_nsxt_id': '995e2516-491f-####-####-f00a5b92ee75',
 'id': '9da07541-4b68-####-####-f51ab26f334e',
 'is_tier0_managed_by_system': True,
 'name': 'mgt-lab-##01',
 'nsxt_cluster_id': 'ee3fe419-32e1-####-####-03f81f4de44f',
 'nsxt_edge_nodes': [{'edge_node_nsxt_id': '6faf7098-800c-####-####-e9b40e1dec8c',
                      'host_name': 'mgt-lab-##01.domain.com',
                      'id': '561ca18a-cfae-####-####-290f40ac67dd',
                      'management_ip_address': '##.##.##.##'},
                     {'edge_node_nsxt_id': 'dbd2aa1b-a227-####-####-64682bcc8d9e',
                      'host_name': 'mgt-lab-##02.domain.com',
                      'id': '9f8b57fc-9ac3-####-####-6ae7bd87a03a',
                      'management_ip_address': '##.##.##.##'}],
 'skip_tep_routability_check': False,
 'status': 'ACTIVE'}, {'cluster_ids': ['0ef703b1-0f66-####-####-07ddc952de60',
                 '31fec2a4-7039-####-####-ddd5fffe1570'],
 'edge_cluster_nsxt_id': 'bb277392-2596-####-####-d626b871df52',
 'id': '361d64ce-1e7e-####-####-4f6a9babc726',
 'is_tier0_managed_by_system': True,
 'name': 'ext-lab-##01',
 'nsxt_cluster_id': '67c58865-0ba8-####-####-c315a0a353d7',
 'nsxt_edge_nodes': [{'edge_node_nsxt_id': '237491f7-3a6c-4####-####-9ebf9787c7e9',
                      'host_name': 'int-##-##01.domain.com',
                      'id': 'a1811058-b75a-4####-####-9c003c512b3c',
                      'management_ip_address': '##.##.##.##'},
                     {'edge_node_nsxt_id': 'a444e70a-b4ac-####-####-e5919cf970e9',
                      'host_name': 'int-##-##02.domain.com',
                      'id': 'aad3c298-897a-####-####-e217a1b497ec',
                      'management_ip_address': '##.##.##.##'}],
 'skip_tep_routability_check': False,
 'status': 'ACTIVE'}, {'cluster_ids': ['ed738207-3113-####-####-5ff8b73e77dd'],
 'edge_cluster_nsxt_id': '6ca4575d-b22b-####-####-a91c1c29bc01',
 'id': '549c17be-edc3-####-####-2065f7a0bedc',
 'is_tier0_managed_by_system': True,
 'name': 'mgt1-lab-##01',
 'nsxt_cluster_id': 'ee3fe419-32e1-####-####-03f81f4de44f',
 'nsxt_edge_nodes': [{'edge_node_nsxt_id': 'cbce4d0d-4307-####-####-35ee0620010b',
                      'host_name': 'mgt-lab-##03.domain.com',
                      'id': '71dc326b-49d2-####-####-c15f39b6927d',
                      'management_ip_address': '##.##.##.##'},
                     {'edge_node_nsxt_id': '51aa2016-c68d-####-####-6b76f9b181f2',
                      'host_name': 'mgt-lab-##04.domain.com',
                      'id': '4bcf28a2-dff6-####-####-6e421ac5888b',
                      'management_ip_address': '##.##.##.###'}],
 'skip_tep_routability_check': False,
 'status': 'ACTIVE'}]

 

• /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

YYYY-MM-DDTHH:MM:SS+0000 VERBOSE [743b6a24ea0c4c90] [services/util.js, http-get-, logRequest:275] Incoming request {"id":"9dd0f7a1-f8ce-####-####-6c45fb64f355","method":"GET","url":"/ui/api/v1/nsxt-clusters"}
YYYY-MM-DDTHH:MM:SS++0000 VERBOSE [4d4c07750b32490a] [services/util.js, http-get-, logRequest:275] Incoming request {"id":"ee97a9d5-8731-####-####-fbe0ad49ebb2","method":"GET","url":"/ui/api/v1/nsxt-clusters"}
YYYY-MM-DDTHH:MM:SS+0000 VERBOSE [40de5dddd3b94a8c] [services/util.js, http-get-, logRequest:275] Incoming request {"id":"41adbe41-754e-####-####-37b9110877d7","method":"GET","url":"/ui/api/v1/nsxt-clusters"}
YYYY-MM-DDTHH:MM:SS+0000 VERBOSE [2c733829a703429f] [services/util.js, http-get-, logRequest:275] Incoming request {"id":"4ee1204b-139b-####-####-95f2578ef7dc","method":"GET","url":"/ui/api/v1/nsxt-clusters"}

• /var/log/vmware/vcf/domainmanager/domainmanager.log:

domainmanager.log:202#-##-06T02:17:35.038+0000 INFO  [vcf_dm,00e5e8105f9c443e,b1e7] [c.v.v.n.s.i.NsxTEdgeClusterServiceImpl,http-nio-127.0.0.1-7200-exec-2]  Obtained list of edge clusters from inventory: [class NsxtEdgeCluster {
domainmanager.log:    nsxtClusterId: ee3fe419-32e1-####-####-03f81f4de44f
domainmanager.log:    edgeClusterNsxtId: 995e2516-491f-####-####-f00a5b92ee75
domainmanager.log:    nsxtEdgeNodes: [class NsxtEdgeNode {
domainmanager.log:        edgeNodeNsxtId: 6faf7098-800c-####-####-e9b40e1dec8c
domainmanager.log:    }, class NsxtEdgeNode {
domainmanager.log:        edgeNodeNsxtId: dbd2aa1b-a227-####-####-64682bcc8d9e

 

Environment

VMware SDDC Manager 5.2.x

Cause

This issue is a resultant of stale data created in Platform database of SDDC Manager due to Out-Of-Band changes made to the SDDC environment.

Example of such scenarios: 

• If you have created a new vSphere cluster within the Workload Domain and/or Management Domain.
• Migrated ESXi hosts along with the VMs from old vSphere cluster to new vSphere cluster and deleted the old cluster.
• After this change you might notice that the "Edge Clusters" tab in the workload domains on SDDC Manager UI is showing Blank entries.
• And Summary Page of each Workload Domain mapped to the Edge Cluster/Nodes will show the Error message: "Failed to get workload domain NSX edge clusters call information" 

Note: In this example, NSX is federated.

Resolution

• Please reach out to Broadcom Global Support team to assist you with identifying whether any stale entry exist in the "Platform" database of SDDC Manager. 
• And if any stale entry exist, Broadcom Global Support team will look for the possibilities to clean up that entry from  "Platform" database.