ESXi host status Unknown: NSX UI opsAgent crash - VMware NSX

Products

• VMware NSX

Issue/Introduction

• In the NSX Manager UI, System > Fabric > Hosts, clicking on Status details shows as below,
  
  Controller Connectivity: Unknown
Tunnel Status: Unknown
PNIC/Bond Status: Unknown
  
  

• Affected ESXi host(s) create opsAgent core dump file(s): /var/core/opsAgent-zdump.###

• During heavy load, multiple ESXi hosts may be affected in batches.

• Data Plane (DFW / VM network communication): not directly affected. No packet loss or network interruption occurs.

• Control & Management Planes: significantly affected. When issue occurs, primarily manifesting as RPC timeouts, TLS handshake delays, risks of component memory leaks, intermittent "Host Not Responding" errors in vCenter or NSX Manager, and sluggish SSH logins.

• /var/run/log/nsx-syslog.log on ESXi host shows similar error,

Wa(###) nsx-opsagent[#######]: NSX ######## - [nsx@4413 comp="nsx-esx" subcomp="opsagent" s2comp="nsx-core" tid="########" level="WARNING"] AsioNsxProvider: (0x##########-2) user dispatcher blocked since 20 seconds

• catalina.log in the following NSX Manager directories: /var/log/proton/, /var/log/cm-inventory/ and /var/log/proxy/ shows similar errors,
  
  WARNING org.bouncycastle.crypto.fips.ContinuousTestingEntropySource getEntropy entropy source stuck
WARNING org.bouncycastle.crypto.fips.ContinuousTestingEntropySource getEntropy proportionate test failed

Environment

• VMware VCF NSX 9.x
• VMware NSX 4.x
• VMware NSX-T Data Center 3.x

Cause

This problem occurs because NSX is requesting high quality entropy for frequent use and consequently the default HWRNG entropy gets exhausted and thereby causes timeouts as we wait for more entropy. Reference ESXi documentation: Controlling ESXi Entropy.

Resolution

This is a known issue impacting VMware NSX and the issue is fixed in ESXi 9.1.1.

Follow below workaround in case environment cannot be upgraded immediately.

Workaround:

Option 1: For temporary issue fix on production host follow below steps.

• SSH to ESXi host as root and run the commands in same order:

/etc/init.d/nsx-opsagent stop

/etc/init.d/nsx-cfgagent stop

/etc/init.d/nsx-proxy stop

/etc/init.d/nsx-nestdb stop

/etc/init.d/nsx-pre-nestdb stop

/etc/init.d/nsx-pre-nestdb start

/etc/init.d/nsx-nestdb start

/etc/init.d/nsx-proxy start

/etc/init.d/nsx-cfgagent start

/etc/init.d/nsx-opsagent start

Option 2: For long-term to reduce the probability of the issue reoccurring:

• 1. Inspect the sources configured to provide entropy:
     
     esxcli system settings kernel list -o entropySources
  2. Enable additional sources of entropy.
     
     esxcli system settings kernel set -s entropySources -v 0xF
     
     Note:
     1. A reboot is required for the ESXi host to use the adjusted configuration.
     2. By implementing this longer term workaround you will lose FIPS compliance

Workaround Reversal:

Post upgrading to 9.1.1 or later, the workaround should be reverted as follows:

esxcli system settings kernel set -s entropySources -v 0

Note: A reboot is required for the ESXi host to use the adjusted configuration.

Additional Information

• The restarting of the service will likely not remediate the issue by themselves.
• The two esxcli system settings kernel commands will correct the issue.

Issue fix updated ESXi 9.1.1.0 release notes