'Initiate guest OS shutdown task' fails with 'A general system error occurred: Invalid fault'

Products

• VMware vSphere ESXi

Issue/Introduction

• 'Initiate guest OS shutdown' task failed with 'A general system error occurred: Invalid fault' after 30 seconds of execution time.

• 'Cannot shut down the guest OS' error event (vim.event.VmFailedToShutdownGuestEvent) can also be seen. 

• In the vmware.log file, you see entries similar to:
  

  ####-##-##T##:##:##.###Z In(05) vmx - Tools: Timed out waiting for state change status.
  ####-##-##T##:##:##.###Z In(05) vmx - Vix: [mainDispatch.c:4123]: VMAutomationReportPowerStateChange: Reporting power state change (opcode=2, err=0).
  ####-##-##T##:##:##.###Z In(05) vmx - VigorTransport_ServerSendResponse opID=######-#####-#####-####-##:########-##-###-###### seq=2805: Completed PowerState request with messages.
  ####-##-##T##:##:##.###Z In(05) vmx - Tools: Changing running status: 2 => 1.

• The .vmx configuration parameter toolScripts.beforePowerOff = "TRUE" is the specific hypervisor
   
• In the hostd.log file, you see entries similar to:
  

  ####-##-##T##:##:##.###Z info hostd[1051735] [Originator@6876 sub=AdapterServer opID=######-#####-#####-####-##:########-##-###-###### user=vpxuser:VSPHERE.LOCAL\Administrator]
  AdapterServer caught exception; <<#######-####-####-####-#########, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 43340'>>, 24, vim.VirtualMachine.shutdownGuest>, N3Vim5Fault20GenericVmConfigFault9ExceptionE(Fault cause: vim.fault.GenericVmConfigFault
  --> )
  --> [context]zKq7AVICAgAAAKjoRgEOaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAZp27WxpYnZpbS10eXBlcy5zbwCBA4oHAYFD5gcBgUagAwECEPNRaG9zdGQAAoY2ywF3ivECSptSAMzHLQBUAy4AAhE/Azt9AGxpYnB0aHJlYWQuc28uMAAEbdEObGliYy5zby42AA==[/context]
  ####-##-##T##:##:##.###Z info hostd[1051735] [Originator@6876 sub=Vimsvc.TaskManager opID=######-#####-#####-####-##:########-##-###-###### user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask-24-vim.VirtualMachine.shutdownGuest-2543346 Status error
  ####-##-##T##:##:##.###Z info hostd[1051735] [Originator@6876 sub=Solo.Vmomi opID=######-#####-#####-####-##:########-##-###-###### user=vpxuser:VSPHERE.LOCAL\Administrator] Activation finished; <<#######-####-####-####-#########, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 43340'>>, 24, vim.VirtualMachine.shutdownGuest>
  ####-##-##T##:##:##.###Z info hostd[1051735] [Originator@6876 sub=Solo.Vmomi opID=######-#####-#####-####-##:########-##-###-###### user=vpxuser:VSPHERE.LOCAL\Administrator] Throw vim.fault.GenericVmConfigFault
  ####-##-##T##:##:##.###Z info hostd[1051735] [Originator@6876 sub=Solo.Vmomi opID=######-#####-#####-####-##:########-##-###-###### user=vpxuser:VSPHERE.LOCAL\Administrator] Result:
  --> (vim.fault.GenericVmConfigFault) {
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "msg.vigor.poweroff.fail.unknown",
  -->          message = "Failed to power off the virtual machine.",
  -->       }
  -->    ],
  -->    reason = "Failed to power off the virtual machine.",
  -->    msg = "Failed to power off the virtual machine.
  --> "
  --> }
  ####-##-##T##:##:##.###Z error hostd[1051735] [Originator@6876 sub=Req@internalvim25... opID=######-#####-#####-####-##:########-##-###-###### user=vpxuser:VSPHERE.LOCAL\Administrator] Method vim.VirtualMachine.shutdownGuest threw undeclared fault of type vim.fault.GenericVmConfigFault

Environment

VMware vSphere ESXi

Cause

Shutdown task fails if VMware Tools custom script for guest OS shutdown takes more than 30 seconds to run.

Resolution

To resolve the timeout conflict depending on the necessity of the internal scripts:

Edit the VM setting and add the following advanced configuration parameter to increase the timeout value.

Preserve Script Execution (Recommended if scripts are required)

Name : tools.statechange.status.timeout 

Value : 300

Edit the .vmx file using vi command and add the following line:

Retain toolScripts.beforePowerOff = "TRUE"

Implement tools.statechange.status.timeout = "300" in the .vmx file to extend the VMX waiting period to 5 minutes, allowing the scripts to finish without triggering a false-positive API failure.

The timeout value should be determined by the time it takes for the custom script to complete.

Additional Information

Tips for editing a .vmx file

Contact Broadcom support