IIAD terminates vCenter Server Service unexpectedly with no indication in vpxd logs

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• The vpxd service crash or restart unexpectedly.
• No vpxd core dumps created at the time of occurrence.
• There may not be an indication of the crash occurrence or backtrace found in the vpxd logs.
• In the iiad log file, you see entries similar to:
  
  Note: iiad logs are in UTC time.
  
  2016-12-24T18:13:21.970Z INFO None
  Error connecting to vCenter Server: _ssl.c:495: The handshake operation timed out
  2016-12-24T18:13:21.993Z INFO Traceback (most recent call last):
  File "D:\Program Files\VMware\vCenter Server\python-modules\iiad\MonitorVpxd.py", line 62, in interrogate rf = si.content.rootFolder
  File "D:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\VmomiSupport.py", line 537, in __call__
  return self.f(*args, **kwargs)
  File "D:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\VmomiSupport.py", line 360, in _InvokeAccessor
  return self._stub.InvokeAccessor(self, info)
  File "D:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\SoapAdapter.py", line 1530, in InvokeAccessor
  self._CallLoginMethod()
  File "D:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\SoapAdapter.py", line 1556, in _CallLoginMethod
  self.loginMethod(self.soapStub)
  .
  .
  2016-12-24T18:13:21.993Z INFO vpxd: service check status: error
  2016-12-24T18:13:21.993Z INFO vpxd: failure count: 2
  2016-12-24T18:42:55.742Z INFO Issuing cmd: ['taskkill', '/F', '/PID', '37048']

 

Environment

VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.0.x

Cause

This issue occurs if IIAD service receives two failed connection attempts, it issues the task kill operation to kill the vpxd service and restart the service.

 

Note: IIAD is essentially a watch dog service. It monitors vCenter Server.

Resolution

To resolve this issue, increase the timeout and maximum failure values to prevent the iiad service from killing the vpxd service.

 

 

To increase the timeout and maximum failure values for iiad:

1. Edit the C:/ProgramData/VMware/vCenterServer/cfg/iiad.json file using a text editor.
2. Modify:
   
   requestTimeout from 20 to 60.
   maxSingleFailures from 2 to 10.
   
   For example:
   
   Here is what the file will look like after these changes:{
   "requestTimeout": 60,
   "hysteresisCount": 4,
   "remediatedHysteresisCount": 6,
   "rebootShellCmd": null,
   "restartShellCmd": null,
   "maxTotalFailures": 50,
   "needShellOnWin": true,
   "watchdogDisabled": false,
   "vpxd.watchdogDisabled": false,
   "createSupportBundle": true,
   "automaticServiceRestart": true,
   "automaticSystemReboot": false,
   "maxSingleRestarts": 3,
   "maxSingleFailures": 10
   }
   
   Note: There is no upper limit to these values. IIAD waits for the configured amount of times before killing the vpxd service. You can have a suitable higher number which stabilizes vpxd.
    
3. Save the file.