SRM Server service crashed during planned failover when at least one of VMs has set recovery.powerOffTimeout=0

Products

• VMware Live Recovery

Issue/Introduction

SRM Server service crashed during planned failover.

According to vmware-dr.log during planned failover, you could see entries like Waiting for VM '[vim.VirtualMachine:b6192ecc-56db-448a-a3fb-2364e4185e2d:vm-1003]' to reach desired powered state 'poweredOff' within '0' seconds., which indicates recovery.powerOffTimeout was 0 on vm-1003.

YYYY-MM-DDThh:mm:ss.###Z verbose vmware-dr[01560] [SRM@6876 sub=Recovery ctxID=49ddcd88 opID=5dddad5d-5b6b-469e-ab1d-fe0eef1458c6-failover:4b4b:bbce:8c49:9666-powerOffVm:f406] WaitForPowerOff: Waiting for VM to shutdown: [vim.VirtualMachine:b6192ecc-56db-448a-a3fb-2364e4185e2d:vm-1003]
YYYY-MM-DDThh:mm:ss.###Z info vmware-dr[01560] [SRM@6876 sub=VmDomain ctxID=49ddcd88 opID=5dddad5d-5b6b-469e-ab1d-fe0eef1458c6-failover:4b4b:bbce:8c49:9666-powerOffVm:f406] Waiting for VM '[vim.VirtualMachine:b6192ecc-56db-448a-a3fb-2364e4185e2d:vm-1003]' to reach desired powered state 'poweredOff' within '0' seconds.
YYYY-MM-DDThh:mm:ss.###Z panic vmware-dr[01560] [SRM@6876 sub=Default ctxID=49ddcd88 opID=5dddad5d-5b6b-469e-ab1d-fe0eef1458c6-failover:4b4b:bbce:8c49:9666-powerOffVm:f406] 
--> 
--> Panic: VERIFY /build/mts/release/bora-24639340/srm/public/functional/async/timedFunc.h:147
--> 
--> Backtrace:
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 9.0.2, build: build-24639340, tag: vmware-dr, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00252482]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x0022B207]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x0048CA7B]
--> backtrace[03] libvmacore.so[0x0048CB82]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x001D310C]
--> backtrace[05] libvmacore.so[0x001D31AA]
--> backtrace[06] libconnection-vc.so[0x004DFEF2]
--> backtrace[07] libconnection-vc.so[0x004E904D]
--> backtrace[08] libconnection-vc.so[0x0048F8AD]
--> backtrace[09] libdr-recovery.so[0x007E28EF]
--> backtrace[10] libdr-recovery.so[0x007E5CE7]
--> backtrace[11] libfunctional.so[0x00039BE1]
--> backtrace[12] libvmacore.so[0x003429CE]
--> backtrace[13] libvmacore.so[0x003442D2]
--> backtrace[14] libvmacore.so[0x00497DE0]
--> backtrace[15] libpthread.so.0[0x00008EB0]
--> backtrace[16] libc.so.6[0x000FFADF]
--> backtrace[17] (no module)
--> [backtrace end]

Environment

VMware Live Site Recovery

Cause

This could happen when at least one VM has recovery.powerOffTimeout=0.

UI rejects setting this value to 0, but we can set it via API.

Resolution

SRM server service crash is unavoidable when at least on VM set recovery.powerOffTimeout=0. Ensure all the VM set none-zero value to recovery.powerOffTimeout. (Default: 300sec)