NSX-T Federation RTEP Tunnels showing degraded

Products

• VMware NSX
• VMware NSX-T Data Center
• VMware NSX Networking

Issue/Introduction

Remote Tunnel Endpoint status shows Degraded or Down after NSX-T Federation environment upgraded from version 3.1.x to 3.2.x.

Environment

VMware NSX-T Data Center, VMware NSX

Cause

This is a known issue occurs when upgrading from NSX 3.1.x to 3.2.x in a Federated environment.

This occurs when one Site T1 gateway does not have the info about remote site edge cluster id. 

In the NSX Manager log located at /var/log/proton/nsxapi.log, the following message may be observed:

YYYY-MM-DDTHH:MM:SS.201Z  INFO l3-tasks1 EdgeClusterMeshUpdateTask 20324 ROUTING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Continue with  full underlay mesh span RemoteMeshSpanForEc [isFullSpan=true, remoteEClusterIds=[XXXXXXXX-XXXX-XXXX-XXXX-XXXX], getRevision()=49545, getIdentifier()=RemoteMeshSpanForEc/XXXXXXXX-XXXX-XXXX-XXXX-XXXX] for edge cluster XXXXXXXX-XXXX-XXXX-XXXX-XXXX

Resolution

Issue a PATCH command to update/add the “description” field for the locale-services of affected T1 for each GM node. This will update the revision counter and push the necessary config. 

• Get list of locale-services on T1 for each GM using below command:
• curl -k -H “Content-Type:application/json” -u admin -X GET https://<NSX_MANAGER_IP>/global-manager/api/v1/global-infra/tier-1s/<tier-1s-id>/locale-services
• Run the PATCH command via curl on Active GM for each locale-service. 
• curl -k -H “Content-Type:application/json” -u admin -X PATCH https://<NSX_MANAGER_IP>/global-manager/api/v1/global-infra/tier-1s/<tier-1s-id>/locale-services/<locale-service-id> -d ‘{“display_name” : “<name_of_Locale_service>”}’