Guest Introspection Status is Warning: Service is not ready

Products

• VMware vDefend Firewall

Issue/Introduction

Symptoms:

• Guest Introspection Status is Warning: Service is not ready.
• Rebooting the Guest Introspection Universal Service VM does not resolve the issue.
• Removing the Guest Introspection Universal Service VM from the inventory and clicking on the Resolve button to recreate does not resolve the issue.

• In the /var/log/syslog.log file of the ESXi host, you see entries similar to:
  
  2017-08-03T08:47:44Z EPSecMux[358242]: [INFO] (EPSEC) [0x57762] Could not identify the USVM for the MuxEventHandler, health status cannot be sent.
  [ ... ]
  2017-08-03T08:48:01Z EPSecMux[358242]: [INFO] (EPSEC) [0x57762] [948959849] Connected to SVM 4219A3EA-5307-6C37-67EE-CCE7388FXXXX : /vmfs/volumes/vsan:52d0ac101a2f0c5d-919ebea82726XXXX/546d6359-1497-ef87-b308-0025b5a3XXXX/Guest Introspection (7).vmx
  2017-08-03T08:48:01Z EPSecMux[358242]: [INFO] (EPSEC) [0x57762] Received new configuration
  2017-08-03T08:48:01Z EPSecMux[358242]: [ERROR] (EPSEC) [0x57762] Failed to find element: vmxPath.
  2017-08-03T08:48:01Z

• In the vsm.log file of the NSX Manager, you see entries similar to:

  2017-08-04 06:42:50.822 GMT ERROR pool-16-thread-1 EndpointConfigurationManagerImpl:1093 - vm vm-268 has null vmxPath for host host-24 - will retry

  Note: Where the vm-id corresponds to the orphaned VM.

• In the NSX Manager logs, you see entries similar to:
  
  2017-08-04 06:42:50.822 GMT ERROR pool-16-thread-1 EndpointConfigurationManagerImpl:1093 - vm vm-268 has null vmxPath for host host-24 - will retry
  2017-08-04 06:42:50.822 GMT ERROR pool-16-thread-1 EndpointConfigurationManagerImpl:1103 - vm has null uuid for host host-24 - will retry
  2017-08-04 06:42:50.844 GMT ERROR pool-16-thread-1 EndpointConfigurationManagerImpl:1093 - vm vm-230 has null vmxPath for host host-18 - will retry
  2017-08-04 06:42:50.844 GMT ERROR pool-16-thread-1 EndpointConfigurationManagerImpl:1103 - vm has null uuid for host host-18 - will retry
  2017-08-04 06:42:50.854 GMT INFO pool-16-thread-1 EndpointConfigurationManagerImpl:1165 - Configuration update rpc messages sent to hosts: [host-25, host-22, host-23, host-24, host-18] - scheduling ret ry to check their status
  2017-08-04 06:42:50.854 GMT INFO pool-16-thread-1 EndpointConfigurationManagerImpl:826 - scheduleConfigurationUpdateCheck: scheduling update check in 60 seconds.
  2017-08-04 06:42:50.855 GMT INFO messagingTaskExecutor-7 EndpointConfigurationManagerImpl:455 - Configuration successfully applied to host host-22.
  2017-08-04 06:42:50.864 GMT ERROR messagingTaskExecutor-4 EndpointConfigurationManagerImpl:451 - Configuration of host host-18 failed: failed to send configuration to mux: Status code: 65538, Message: "Error - Mux Parsing config"

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.2.x

Cause

This issue occurs when there are orphaned virtual machines on the ESXi hosts. You can see the orphaned virtual machines in the vSphere Web Client and they appear as Powered Off with Status of Unknown.

Resolution

To resolve this issue, remove the orphaned virtual machines from the vCenter Server inventory. 

 

Note: It takes about 1-2 minutes for the Guest Introspection status to change back to Green.