A connectivity issue has been observed with the nsx-node-agent to NSX following a TPCF upgrade, specifically indicating 'no connection.hyperbus service active'

Products

• VMware NSX
• VMware Tanzu Application Service

Issue/Introduction

• NSX node-agent (which is installed within the Diego cell VM) throws error NCP01012 stating "Agent is exiting as connection is unavailable" . The following log snippet is observed under /var/log/nsx-ujo/nsx_node_agent.log.


<Time-stamp> #######-####-####-####-############ NSX 120188- [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="WARNING"] nsx_ujo.agent.agent Agent is unavailable for 30 seconds: connection inactive.hyperbus service inactive., retrying
<Time-stamp> #######-####-####-####-############ NSX 120188- [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="ERROR" errorCode="NCP01012"] nsx_ujo.agent.agent Agent is exiting as connection is unavailable

• Communication between Diego cell VM and the hyperbus(vmk50) of the host where the Diego cell VM is hosted is broken. 
• NCP looks stable from NSX manager UI, navigate to System > Fabric > Nodes > Container Clusters) to verify. 
  
  
• Hyberbus connection info output on the ESXi host where the problematic Diego Cell VM is hosted looks healthy.
  
• A packet capture on the ESX host’s switchport interfaces (VnicTx and VnicRx) for the affected Diego cell VM indicates that no traffic is being transmitted towards the vmk50 MAC address.
• Similarly, packet captures taken on the vmk50 interface of the same ESX host confirm that no packets are being received from the vNIC MAC address of the Diego cell VM.

 

Environment

VMware NSX

Cause

The issue was triggered by the recent upgrade of Tanzu Platform for Cloud Foundry (TPCF) from version 6.0.12 to 6.0.14. This behavior is documented in this article.

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.