HCX Service mesh upgrade or fleet appliance redeployment fails with following error, "Insufficient capacity on each physical CPU"

Products

• VMware HCX

Issue/Introduction

• Following are symptoms observed under HCX manager logs, /common/logs/admin/app.log

<Timestamp> [InterconnectService_SvcThread-43618] ERROR c.v.v.h.s.i.UpdateApplianceReservation-  Updating appliance reservation failed for user <username>. Reason: Insufficient capacity on each physical CPU. id : host-<host-ID> Caused by: com.vmware.vim.binding.vim.fault.InsufficientPerCpuCapacity: Insufficient capacity on each physical CPU.

• CPU and Memory reservation was configured as 100 in the compute profile.
  

Environment

VMware HCX

Cause

Insufficient cluster CPU capacity. The requested resource reservation exceeded the available physical CPU and memory overhead currently provisioned in the environment.

Resolution

 To resolve the issue:

1. Refer this article to change CPU and memory reservations to 0 and save the updated compute profile.

   1. • Setting the CPU reservation below 90% will usually succeed. It all depends on the available resources on the ESXi hosts throughout the cluster. 
        
        The best way to see what resources are available is to view the "CPU Reservation Details" in vCenter.

        • In VC UI click on the cluster --> Monitor --> Under "resource Allocation" click on CPU
        • The "available Reservation" value is for the entire cluster. Divide that by number of hosts. 

        To pull the exact reservation available for any given host you can navigate to the host MOB. 

        • https://ESXi_FQDN/mob/?moid=ha-root-pool&doPath=config.cpuAllocation
          • NOTE: Please substitute actual hostname in the above. 
        • Host MOB may be disabled. To enable create and/or set the below value to true:
          • Config.HostAgent.plugins.solo.enableMob          true 

        HCX uses the below calculation to set reservation:

        CPU reservation: 8 * 2000 * 100/100 = 16000

        8= number of Logical Processers on ESXi host

        2000= Processor Speed in MHZ 

        100/100 = Reservation value over max reservation (100)

        If reservation set to 95% it would be 95/100

        Workaround:

        In most cases setting the reservation value to <90% should allow deployment to complete. 
        
        OR 
        
        Leave reservation set to 0

2. Perform: Service mesh Resync