Edge and Transport Nodes shows "UNKNOWN" and "Not available" status

Products

• VMware NSX

Issue/Introduction

• The status for the random edge tunnel shows as "Not available," and the Transport Nodes status is listed as "Unknown," as indicated in the screenshots below.

Edge Node:

Transport Node:

• The controller connectivity always shows connected shown in the below.

[root@esxi:~] nsxcli -c get controllers
Wed Sep 18 2024 UTC 06:10:42.545
 Controller IP    Port     SSL         Status       Is Physical Master   Session State  Controller FQDN
  x.x.224.42     1235   enabled      not used            false              null              NA
  x.x.224.40     1235   enabled     connected             true               up               NA
  x.x.224.41     1235   enabled      not used            false              null              NA

• The connectivity on the required ports between the edge and the Manager, as well as between the transport node and the Manager, appears to be functioning well based on the output from the nc -z command.
• The API response will give the same status as unknown for the affected nodes as shown in the below:

{
  "node_uuid" : "dcba5a9d-2f0c-4e6d-8dc4-XXXX",
  "node_display_name" : "XXXXX",
  "status" : "UNKNOWN",
  "pnic_status" : {
    "status" : "UNKNOWN",
    "up_count" : 0,
    "down_count" : 0,
    "degraded_count" : 0
  },
  "mgmt_connection_status" : "UP",
  "control_connection_status" : {
    "status" : "UNKNOWN",
    "up_count" : 0,
    "down_count" : 0,
    "degraded_count" : 0
  },
  "tunnel_status" : {
  "status" : "UNKNOWN",
    "up_count" : 0,
    "down_count" : 0
  },

  "node_status" : {
    "last_heartbeat_timestamp" : 1712423518181,
    "mpa_connectivity_status" : "UP",
    "mpa_connectivity_status_details" : "Client is responding to heartbeats",
    "lcp_connectivity_status" : "UNKNOWN",
    "lcp_connectivity_status_details" : [ ],

• The NSX manager log var/log/proton/nsxapi.log has a log similar to example:

nsxapi.log:2024-09-19T07:43:04.192Z  INFO UfoIndexer-search_manager-0 AggTnStatusQueriesImpl 4305 MONITORING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] node dcba5a9d-2f0c-4e6d-8dc4-065119f9d803 heartbeat timeout, current 1726731784192, ccp 1726731358757, interval 360000 in milliseconds, isExpired:true

 

nsxapi.log:2024-09-19T08:09:29.534Z  INFO http-nio-127.0.0.1-7440-exec-982 AggTnStatusQueriesImpl 4305 MONITORING [nsx@6876 comp="nsx-manager" level="INFO" reqId="fafd3bfe-3b9d-4753-8818-a2e9ee99b2b1" subcomp="manager" username="UC"] node dcba5a9d-2f0c-4e6d-8dc4-065119f9d803 heartbeat timeout, current 1726733369534, ccp 1726733009218, interval 360000 in milliseconds, isExpired:true

Environment

VMware NSX

Cause

In the nsxapi.logs, the heartbeat timestamp 'ccp 1726733009218' is on the rise, yet it still indicates "isExpired: true." This suggests that there is a consistent gap between the current time and the heartbeat time.

It's quite possible that the system times are not synchronized between the Manager nodes and the Transport Nodes.

Resolution

To resolve this issue, validate and correct the system time on both the NSX Manager and the Transport Nodes.