Controller service on Manager node <Manager-ID> to Transport node <Edge-ID> down for at least 15 minutes from Controller service's point of view remains OPEN after upgrading to NSX 4.2.2 or later

Products

• VMware NSX

Issue/Introduction

• After upgrading VMware NSX from a version prior to 4.2.2 (e.g., 4.2.1.x) to version 4.2.2 or later, the following alarm remains permanently in an OPEN state in the NSX Manager UI:

  • Alarm Event Name: control_channel_to_transport_node_down_long

  • Alarm Description: Controller service on Manager node <Manager-ID> to Transport node <Edge-ID> down for at least 15 minutes from Controller service's point of view.

• The underlying Transport/Edge node is fully connected, healthy, and operating normally

Environment

VMwareNSX

Cause

During an upgrade from any version lower than 4.2.2 to version 4.2.2 or later, the Central Control Plane changes its Alarm event source ID format from nsx_ccp to nsx_ccp_35.

If a Controller channel Down alarm was active prior to initiating the upgrade, the post-upgrade system cannot match the incoming recovery event with the pre-upgrade alarm ID due to this source ID format mismatch. Consequently, the recovery event fails to automatically clear the existing alarm, leaving it stuck in an OPEN state.

Resolution

This issue is fixed for future upgrades. Upgrades originating from NSX 4.2.2 or later will not encounter this issue, as the event source ID format will already be standardized.

If you are experiencing this issue post-upgrade, perform the following steps:

1. 1. Verify Transport Node Health: Confirm that the Edge / Transport Node is operating normally by performing standard health checks 

      1. Port Verification: From the affected Edge/Hosts, test connectivity to all your NSX managers on the required ports:

         nc -zv <manager-ip> 1234
         nc -zv <manager-ip> 1235
         

      2. Host-Level Connection State: Run this command directly from the CLI of the ESXi host hosting the affected Edge to check active socket mappings:

         esxcli network ip connection list
         

      3. NSX Component Status: Execute these commands from the ESXi host CLI to check live connection states directly from the NSX agent layer:

         nsxcli -c get controllers
         nsxcli -c get managers
         

      4. Packet Captures: If possible, initiate temporary packet captures on the management switchports of both the flapped Edge and all three NSX Managers to capture any dropped packets or transmission retries during the event window.

   2. Once node operational status is confirmed,Manually Clear the Alarm.

      • Log into the NSX Manager UI.

      • Navigate to Alarms.

      • Locate the stuck control_channel_to_transport_node_down alarm.

      • Manually Acknowledge and Resolve/Clear the open alarm.