NSX Mux Health Failure messages with Guest Introspection installed

Products

• VMware NSX

Issue/Introduction

Symptoms:

• When Mux is not able to get health stats timely due to host being busy working on VM operations like vMotion / power on and off / new VMs bring up, you might see the error on vsphere UI “Lost Communication with ESX module” as shown:
  
  
   

• On the GI VM hosts, you might see error messages on syslog.log like: 

  2018-09-15T01:45:26Z EPSecMux[695598]: [ERROR] (EPSEC) [999999] popen failed for pidof NSX-Context-Engine. Errno: 28 (errno is not set on allocation failures). Strerror: No space left on device.

  2018-09-15T01:45:26Z EPSecMux[695598]: [ERROR] (EPSEC) [999999] Exception encountered while querying Context Engine status, EPSecPosixError@tid=999999: popen failed. errno: 28 (No space left on device)

  2018-09-15T01:45:13Z EPSecMux[695598]: [ERROR] (EPSEC) [695598] Attempted to recv 46 bytes from sd 72, errno = 12 (Cannot allocate memory)

  2018-09-15T01:45:13Z EPSecMux[695598]: [ERROR] (EPSEC) [695598] [1751854430] Error on guest socket: SocketError on sd 72, in recv: Cannot allocate memory (12)

  2018-09-15T01:44:49Z EPSecMux[488907]: [ERROR] (EPSEC) [488907] read_key Namespace database operation failed for key XXXXXXXX-9999-99XX-XXX9-XXXXXXX9999: No namespace found

  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.4.x

Cause

The issue is seen when the host is busy processing many VM operations like bulk vMotion or bulk VM power off/on operations.

 These are benign messages and shouldn’t cause any AV functionality from a GI perspective.

Resolution

Host busy processing would normally resolve itself.   

If you see this frequently and the error message does not resolve itself or if there is any AV functionality impact. Contact Broadcom Support.