Transport node in status is inactive in the NSX-T GUI and opsagent service is not running at the ESXi level

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

• Transport node in status is  inactive in the NSX-T GUI.
• opsagent service is not running at the ESXi level.
• opsagent core dumps present in /var/core on the ESXi host
• ESXi Host fails to enter Maintenance Mode due to VMs not getting migrated.
• In  var/log/vobd.log, following log lines are observed:

[UserWorldCorrelator] #######693137us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(#######) /var/core/opsAgent-zdump.003
[UserWorldCorrelator] #######275029us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(#######) /var/core/opsAgent-zdump.000
[UserWorldCorrelator] #######468259us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(#######) /var/core/Ops:Alarm-zdump.000
[UserWorldCorrelator] #######340419us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(#######) /var/core/opsAgent-zdump.001

• In var/log/nsx-syslog.log, following log lines are observed:

nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="mpa-client" tid="#######" level="INFO"] [AlarmsProvider] CreateAppSpecificInstance End (MPA_CLIENT_PURGE_TIMER: 300000)
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="mpa-client" tid="#######" level="INFO"] [AlarmsProvider] Starting MPA client thread...
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="tnproxy-client" tid="#######" level="INFO"] [AlarmsProvider] TnProxyClientCommon SetTnProxyConnectionUp
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="tnproxy-client" tid="#######" level="INFO"] [AlarmsProvider] TnProxyClientCommon SetTnProxyConnectionDown
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="tnproxy-client" tid="#######" level="INFO"] [AlarmsProvider] TnProxyClientCommon SetUpdateStubs
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="mpa-client" tid="#######" level="WARNING"] [AlarmsProvider] connecting to mpa return cookie [NO_SESSION]
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="alarmsprovider" tid="#######" level="INFO"] Starting Client Library Communicator
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="alarmsprovider" tid="#######" level="INFO"] Starting Provider Services
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="nsx-opsagent" s2comp="upm" tid="#######" level="INFO"] Nestdb channel status turns to DOWN
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="nsx-opsagent" s2comp="upm" tid="#######" level="INFO"] started
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="#######" level="INFO"] Waiting for Controller List app start..
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="#######" level="INFO"] [ReadData] fd [29] totalByteRead = 27 err = [ : 0]
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="#######" level="INFO"] [TerminateVimPy] retry terminated prior nsxVim.py process: [process [ 6551117 ] killed ]
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="#######" level="INFO"] [ReadData] fd [19] totalByteRead = 64 err = [ : 0]
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="#######" level="INFO"] [ReadData] fd [19] totalByteRead = 17 err = [ : 0]
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="#######" level="INFO"] [SpawnPyVimInternal] Spawning python /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py 2> /var/log/nsxaVim.err....
nsx-opsagent[#######] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="#######" level="INFO"] [ReadData] fd [29] totalByteRead = 0 err = [ : 0]

Environment

VMware NSX-T Data Center

Cause

The opsagent service is crashing due to an issue in context engine.

Resolution

This issue is resolved in NSX-T version 3.1.2

Workaround:
If not using IDFW/IDS/NSX Intelligence in the environment, on each of the affected hosts, disable context Engine as instructed below:

 

• On an affected ESXi host,
  
  Go to vi /etc/vmware/nsx-opsagent/nsx-opsAgent.xml and change the line
  
  From
  <ctxteng>true</ctxteng>
  
  To
  <ctxteng>false</ctxteng>
  
  Save the changes.

 

• Restart nsx-opsagent:
  
  /etc/init.d/nsx-opsagent restart
  
  

 

NOTE: No further host reboot is required following above steps

NOTE: IDFW is Identity Firewall, IDS is Intrusion detection system