Virtual machines including Guest Introspection freeze with connectivity issues

Products

• VMware NSX

Issue/Introduction

Symptoms:

• In the NSX Manager GUI, alarms can be seen such as:
  Event Message: Lost communication with ESX Module.
   
• In the NSX Manager logs, below log lines can be seen:
  
  management_service/vsm.log.4:2017-04-04 01:12:48.039 CEST INFO pool-9-thread-1 SystemEventDaoImpl:134 - [SystemEvent] Time:'Tue Apr 04 01:12:48.035 CEST 2017', Severity:'Informational', Event Source:'Guest Introspection Health Status', Code:'260007', Event Message:'Lost communication with ESX module.', Module:'Guest Introspection Health Status', Universal Object:'false'

management_service/vsm.log.4:2017-04-04 01:32:48.024 CEST INFO pool-9-thread-1 SystemEventDaoImpl:134 - [SystemEvent] Time:'Tue Apr 04 01:32:48.019 CEST 2017', Severity:'Informational', Event Source:'Guest Introspection Health Status', Code:'260007', Event Message:'Lost communication with ESX module.', Module:'Guest Introspection Health Status', Universal Object:'false'

 

• In the /var/run/log/vobd.log file, below entries or similar entries can be seen:
  
  2017-04-10T02:55:03.688Z: [scsiCorrelator] 5378664846910us: [esx.problem.scsi.device.io.latency.high] Device naa.60050768################## performance has deteriorated. I/O latency increased from average value of 3110 microseconds to 68131 microseconds.

  2017-04-10T02:57:10.735Z: [scsiCorrelator] 5378805293434us: [vob.scsi.device.io.latency.improved] Device naa.60050768##################### performance has improved. I/O latency reduced from 68131 microseconds to 13447 microseconds.

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x

Cause

This issue occurs due to underlying storage latency issues.

Resolution

To resolve the issue, fix the storage issues in the environments due to which the VMs are having problems read/writing to disk.