vCenter crashes after enabling SDRS

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter crashes on enabling SDRS.
• In the vpxd.log file, you see entries similar to:
  
  2016-12-05T15:55:48.122-05:00 panic vpxd[10860] [Originator@6876 sub=Default opID=vcd-########-####-####-####-########dd19;activity=urn:uuid:########-####-####-####-########9bb9-4e]
  -->
  --> Panic: Win32 exception: Access Violation (0xc0000005)
  --> Read (0) at address 0000000000000028
  --> rip: 00007ff743d1ebd8 rsp: 000000001b62c140 rbp: d51a5204841da711
  --> rax: 0000000000000112 rbx: 0000000000000000 rcx: 46e9b06a451e8b12
  --> rdx: 000000001035abd0 rdi: 0000000000000050 rsi: 0000000000000111
  --> r8: 000000001f59e100 r9: 0000000018548228 r10: 000000001ffc9030
  --> r11: 0000000018548218 r12: 000000001f59e0c0 r13: 0000000018547240
  --> r14: 0000000000000001 r15: 0000000000000001
  -->
  --> Backtrace:
  -->
  --> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-3634793, tag: vpxd
  --> backtrace[00] vmacore.dll[0x001C599A]
  --> backtrace[01] vmacore.dll[0x0005C8BF]
  --> backtrace[02] vmacore.dll[0x0005DA0E]
  --> backtrace[03] vmacore.dll[0x001D7405]
  --> backtrace[04] vmacore.dll[0x001D7546]
  --> backtrace[05] vmacore.dll[0x001DAC66]
  --> backtrace[06] MSVCR90.dll[0x00045129]
  --> backtrace[07] MSVCR90.dll[0x000442D6]
  --> backtrace[08] MSVCR90.dll[0x000449DE]
  --> backtrace[09] MSVCR90.dll[0x00044C1D]
  --> backtrace[10] MSVCR90.dll[0x0004502B]
  --> backtrace[11] vpxd.exe[0x013CAD32]
  --> backtrace[12] ntdll.dll[0x000933FD]
  --> backtrace[13] ntdll.dll[0x00054847]
  --> backtrace[14] ntdll.dll[0x0009258A]
  --> backtrace[15] vpxd.exe[0x008AEBD8]

Environment

VMware vCenter Server 6.0.x

Cause

This issue is causing because of pbmcalls.

Resolution

This issue is resolved in: 
vCenter Server 6.0 Update 3c | 9 NOV 2017 | Build 7040641
vCenter Server 6.5 

To workaround this issue, increase the timeout and maximum failure values to prevent the iiad service from stopping the vpxd service.

Follow steps below, to increase the timeout and maximum failure values for iiad:

1. Edit the C:/ProgramData/VMware/vCenterServer/cfg/iiad.json file using a text editor.
2. Modify:
   requestTimeout from 20 to 60
   maxSingleFailures from 2 to 10
   
   Sample Output, here is how the file will look after the changes:
   {
   "requestTimeout": 60,
   "hysteresisCount": 4,
   "remediatedHysteresisCount": 6,
   "rebootShellCmd": null,
   "restartShellCmd": null,
   "maxTotalFailures": 50,
   "needShellOnWin": true,
   "watchdogDisabled": false,
   "vpxd.watchdogDisabled": false,
   "createSupportBundle": true,
   "automaticServiceRestart": true,
   "automaticSystemReboot": false,
   "maxSingleRestarts": 3,
   "maxSingleFailures": 10
   }
   
   Note: There is no upper limit to these values. IIAD waits for the configured amount of times before stopping the vpxd service. You can have a suitable higher number which stabilizes vpxd.
    
3. Save the file.

Additional Information

Looking at the previous call for the same OpId before the crash is:
------------------------------------------------------------------------
2016-12-05T15:53:36.971-05:00 info vpxd[04576] [Originator@6876 sub=pbm opID=vcd-########-####-####-####-########dd19;activity=urn:uuid:########-####-####-####-########5744-3f-e6] PBMCallback: PbmFunctionTracer::PbmFunctionTracer: Entering PbmCallBack::PostProcessReconfigureSpec
2016-12-05T15:53:36.971-05:00 info vpxd[04576] [Originator@6876 sub=pbm opID=vcd-########-####-####-####-########dd19;activity=urn:uuid:########-####-####-####-########5744-3f-e6] PBMCallback: PbmCallBack::ReconfigVmAssociations: Reconfigure profile association for vm vm-118051
2016-12-05T15:53:36.971-05:00 info vpxd[04576] [Originator@6876 sub=pbm opID=vcd-########-####-####-####-########dd19;activity=urn:uuid:########-####-####-####-########5744-3f-e6] PBMCallback: PbmFunctionTracer::PbmFunctionTracer: Entering PbmProfileManagerProxy::ReconfigVmAssociations