BOSH Director Tasks hang intermittently during Apply Changes

Products

Operations Manager

Issue/Introduction

BOSH Director tasks hang during Apply Changes while waiting for CPI calls to vCenter to complete. There's no timeout for the call and BOSH expects the CPI to exit but it doesn't. This is due to a lack of resources in the vSphere API client. The vAPI process on the vSphere server is limited to 256MB of memory by default.

To confirm, look for 'outofmemory' errors in the following log: \var\log\vmware\vapi\endpoint.log

Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.List]: Factory method 'getComponents' threw exception; nested exception is java.lang.OutOfMemoryError: Java heap space

Environment

Product Version: 2.8

Resolution

Temporary Workaround

Reboot the director VM which resumes the job and eventually completes successfully.

Permanent Fix

vAPI Endpoint - provides a single point of access to vAPI services

Note: Service names may vary between vCenter Server versions 6.0, 6.5 and 6.7, as such please see How to Stop, Start or Restart vCenter Server 6.x Services (2109881 ) for service names and on how to stop and start services in Windows Center Server and vCenter Server Appliance.

To increase the memory of individual vCenter Server components including 'vmware-vapi-endpoint' please follow the steps bellow.

1. vCenter Server Appliance:

Use SSH to connect to the VCSA and enter the BASH shell.
Run the following command to output memory assigned to each of the services:

cloudvm-ram-size -l

root@paisasvc01 [ ~ ]# cloudvm-ram-size -l
vmware-imagebuilder  =       50
vmware-trustmanagement =       78
vmware-analytics     =      218
vmware-netdumper     =       20
OS                   =     1862
vmware-sps           =     1566
vmware-stsd          =      952
vmware-certificatemanagement =       78
vmdnsd               =       39
vmware-vpostgres     =     2018
vsphere-ui           =     1454
vmware-vsan-health   =      106
vmafdd               =       39
vmware-mbcs          =      128
vsan-dps             =      258
vmware-sts-idmd      =      384
vmware-vpxd-svcs     =     1195
vmonapi              =       15
vmware-vsm           =      201
vmware-vapi-endpoint =      256
vmware-cm            =      353
vmware-vpxd          =     7670
vmdird               =       39
vmware-cis-license   =      192
vmware-updatemgr     =      484
vcha                 =       46
vmware-rbd-watchdog  =      100
vmware-pod           =      100
vmware-hvc           =       68

2. Increase the amount of memory assigned to 'vmware-vapi-endpoint' by running the following command: cloudvm-ram-size -C XXXX vmware-vapi-endpoint

Note: XXXX should be replaced by the desired amount of memory in MB.

For example, cloudvm-ram-size -C 2048 vmware-vapi-endpoint would increase the memory of the vmware-vapi-endpoint service from 768 MB to 2048 MB.

Where there is no one size fits all memory value to set in this scenario, as a guide, double the size of currently assigned memory of the service that has an issue. This may need to be repeated.

3. Restart the service that has been altered, in this example, vmware-vapi-endpoint:

service-control --stop vmware-vapi-endpoint;service-control --start vmware-vapi-endpoint
Confirm that the service has now been allocated the increased memory using the command: cloudvm-ram-size -l