"Guest Introspection Service is not ready" error in Guest Introspection service deployment

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX Manager is unable to reach the USVM on the host.
  
  
• Failed to establish a connection between MUX IP and the host on which USVM is installed and muxconfig.xml file is missing in the host.
  
  
• In the syslog.log file of the ESXi host, you see entries similar to:
  
  2016-12-21T04:00:49Z EPSecMux[11410628]: [INFO] (EPSEC) [0xae1cc4] vShield Endpoint Multiplexor build-2959573. [6.0.0]
  2016-12-21T04:00:49Z EPSecMux[11410628]: [INFO] (EPSEC) [0xae1cc4] VMkernel vxln2pesx0003.wpp.net 5.5.0 #1 SMP Release build-3568722 Feb 18 2016 16:42:56 x86_64
  2016-12-21T04:00:49Z EPSecMux[11410628]: [INFO] (EPSEC) [0xae1cc4] Could not identify the USVM for the MuxEventHandler, health status cannot be sent.
  2016-12-21T04:00:50Z vShield-Endpoint-Mux: vShield-Endpoint-Mux started
  
  
• In the NSX Manager vsm.log file, you may see entries similar to:
  
  2016-12-09 06:42:13.782 GMT+00:00 ERROR taskScheduler-11 ClientServiceImpl:280 - Messaging system credentials verification failed. Username = 'usvm-admin-host-56'. com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:533)
  
  
• In the USVM.log file on the Service VM, you may see entries similar to:
  
  2016-12-21 04:07:24.009 UTC ERROR main CustomConnectionFactory:89 - IO Error occurs when creating framechandler, typically a connection issue : java.net.ConnectException: Connection timed out
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.2.x

Cause

This issue occurs due to network issues between the NSX Manager and the USVM deployed on the host. The USVM is unable to contact the NSX Manager to get the config file.

Resolution

To resolve the issue, fix the network connectivity issues between NSX Manager and the USVM before performing redeploy of the GI service on the Cluster.