"insufficient resources to satisfy configured failover level for vSphere HA" error and BOSH tasks hang during Apply Changes

Products

• Operations Manager

Issue/Introduction

BOSH tasks hang during Apply Changes and produce the following error when looking at the details of a particular hung task.

L Error: Unknown CPI error 'Unknown' with message 'Could not power on VM '<[Vim.VirtualMachine] vm-128883>': Insufficient resources to satisfy configured failover level for vSphere HA.' in 'crPI request ID: 'cpi-839390') Task 164775 | 16:35:54 | Error: Error resolving problem '3153': Unknown CPI error 'Unknown' with message 'Could not power on VM '<[Vim.VirtualMachine] vm-128883>': Insufficient resources to satisfy configured failovA.' in 'create_vm' CPI method (CPI request ID: 'cpi-839390')

Note: If you have a running deployment, it is not recommended to cancel queued tasks. It may result in a broken deployment.

To see all currently active tasks, run this command:

bosh tasks

To get more details, run these commands:

bosh task <num> --cpi > task-<num>-cpi.log
bosh task <num> --debug > task-<num>-debug.logs  

Admission control is a policy used by vSphere High Availability (HA) to ensure failover capacity within a cluster. 

This issue occurs when the HA configuration of CPU or memory resources reserved for failover is violated or is insufficient to perform the intended task.

Symptoms include:

• Apply Changes or upgrades hang
• BOSH tasks hang
• BOSH tasks timing out or failing to complete
• Queued tasks piling up

Environment

Product Version: 2.10

Resolution

In the vSphere Console, navigate to the cluster experiencing the issue and do the following: Configure > vSphere Availability > Edit vSphere HA > Admission Control > Disable Host Failover Capacity or Modify the Reserved Capacity

This allows the tasks to resume processing and the Apply Changes or upgrade to continue.