Monitoring Service Framework Health Check on Transport node Alarm observed due to nsx-fabric-exporter block

Products

• VMware NSX

Issue/Introduction

• The NSX monitoring framework reports an unhealthy status and generates an alarm on an affected host.This condition is continuously present and does not correlate with system load or scale.
• When reviewing the nsx-syslog files, the nsx-fabric-exporter subcomponent repeatedly logs a blocked user dispatcher thread warning:
  [TIMESTAMP] Wa(180) nsx-exporter[PID]: NSX PID - [nsx@6876 comp="nsx-esx" subcomp="nsx-fabric-exporter" s2comp="nsx-core" tid="TID" level="WARNING"] AsioNsxProvider: user dispatcher 0xXXXXXXXXXX **blocked since 30 seconds**
• Even after restarting the nsx-exporter service, the thread blocking alerts reappear in the logs within two minutes.
• Additionally, attempting to manually run the following troubleshooting command directly on the affected host results in an indefinite hang condition:
  /sbin/localcli system process stats load get
• Other commands that also require disk / storage related reads may also not complete successfully such as 'df -h'.
• This indicates that the execution block occurs at the system level and is independent of the nsx-exporter daemon itself.
• Dell PowerPath Module in use on the host.
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX 

Cause

• The nsx-exporter service hangs because it is blocked waiting to receive data from the local command (/sbin/localcli system process stats load get). The localcli process itself is indefinitely stalled at the kernel layer, waiting on a storage level resource call.
• Previous occurences were scoped to a resource lock within the third-party Dell PowerPath (emcp) driver module however this has the potential to occur due to storage failures without specific vendor application.

Resolution

• This issue is caused by a third-party software component. Broadcom does not have visibility or code control over the internal locking mechanisms and memory management of the third-party EMCP module.
• If this issue is observed then please engage Dell Technical Support to investigate the kernel-level hang condition within Dell PowerPath and / or your storage teams for storage health issues.