Transport node in status is inactive in the NSX-T GUI and opsagent service is not running at the ESXi level
search cancel

Transport node in status is inactive in the NSX-T GUI and opsagent service is not running at the ESXi level

book

Article ID: 318554

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware NSX

Issue/Introduction

To resolve the opsagent service.

  • Transport node in status is  inactive in the NSX-T GUI.
  • opsagent service is not running at the ESXi level.
  • You notice opsagent core dumps in /var/core dir of the ESXi host
  • ESXi Host failed to enter Maintenance Mode  due to VMs are not getting migrated.
  • In  var/log/vobd.log, you see similar to below

2021-05-26T18:26:01.488Z: [UserWorldCorrelator] 1283606693137us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(6549157) /var/core/opsAgent-zdump.003
2021-05-26T18:26:21.071Z: [UserWorldCorrelator] 1283626275029us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(6550012) /var/core/opsAgent-zdump.000
2021-05-26T18:26:31.264Z: [UserWorldCorrelator] 1283636468259us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(6550627) /var/core/Ops:Alarm-zdump.000
2021-05-26T18:26:51.137Z: [UserWorldCorrelator] 1283656340419us: [vob.uw.core.dumped] /usr/lib64/vmware/nsx-opsagent/bin/opsAgent(6551009) /var/core/opsAgent-zdump.001

  • In var/log/nsx-syslog.log, you see similar to below

2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="mpa-client" tid="6551651" level="INFO"] [AlarmsProvider] CreateAppSpecificInstance End (MPA_CLIENT_PURGE_TIMER: 300000)
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="mpa-client" tid="6551651" level="INFO"] [AlarmsProvider] Starting MPA client thread...
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="tnproxy-client" tid="6551651" level="INFO"] [AlarmsProvider] TnProxyClientCommon SetTnProxyConnectionUp
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="tnproxy-client" tid="6551651" level="INFO"] [AlarmsProvider] TnProxyClientCommon SetTnProxyConnectionDown
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="tnproxy-client" tid="6551651" level="INFO"] [AlarmsProvider] TnProxyClientCommon SetUpdateStubs
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="mpa-client" tid="6551651" level="WARNING"] [AlarmsProvider] connecting to mpa return cookie [NO_SESSION]
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="alarmsprovider" tid="6551651" level="INFO"] Starting Client Library Communicator
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="alarmsprovider" tid="6551651" level="INFO"] Starting Provider Services
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="nsx-opsagent" s2comp="upm" tid="6551651" level="INFO"] Nestdb channel status turns to DOWN
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="nsx-opsagent" s2comp="upm" tid="6551651" level="INFO"] started
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="6551651" level="INFO"] Waiting for Controller List app start..
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="6551665" level="INFO"] [ReadData] fd [29] totalByteRead = 27 err = [ : 0]
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="6551665" level="INFO"] [TerminateVimPy] retry terminated prior nsxVim.py process: [process [ 6551117 ] killed ]
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="6551680" level="INFO"] [ReadData] fd [19] totalByteRead = 64 err = [ : 0]
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="6551680" level="INFO"] [ReadData] fd [19] totalByteRead = 17 err = [ : 0]
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="6551680" level="INFO"] [SpawnPyVimInternal] Spawning python /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py 2> /var/log/nsxaVim.err....
2021-05-26T18:26:51Z nsx-opsagent[6551651] NSX 6551651 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="6551680" level="INFO"] [ReadData] fd [29] totalByteRead = 0 err = [ : 0]


Note: The preceding log excerpts are only examples. Date,time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

The opsagent is crashing due to an issue in context engine.

Resolution

This issue is resolved in NSX-T version 3.1.2

Workaround:
If you are not using IDFW/IDS/NSX Intelligence in their environment, on each of the affected hosts, disable context Engine as instructed below:

NOTE: IDFW is Identity Firewall, IDS is Intrusion detection system

1. On an affected ESXi host,

Go to vi /etc/vmware/nsx-opsagent/nsx-opsAgent.xml

and change the line

From
<ctxteng>true</ctxteng>

To
<ctxteng>false</ctxteng>

Save the changes.

2. Restart nsx-opsagent using

/etc/init.d/nsx-opsagent restart

Note: No further host reboot is required following the above steps


Additional Information

Impact/Risks:
None

Powered by Wolken Software