Upgrade of tile VMs fails with "Response exceeded maximum allowed length"

Products

• Support Only for Spring

Issue/Introduction

During the upgrade of a tile, for example, the Spring Cloud Services, new packages, logs, and releases need to be added to the VM under the /var/vcap/data path. 

If there is not enough space to allocate these new resources the deployment or Apply Changes fails with the following error: 

task 983507 | 11:44:43 | Updating instance spring-cloud-services: spring-cloud-services/27386766-2ae3-4fd2-b744-ecd3d8666df2 (0) (canary) (00:00:14)
Updating deployment:
Expected task '983507' to succeed but state is 'error'
Exit code 1
L Error: Response exceeded maximum allowed length

The Debug logs also show RpcRemoteException errors:

D, [2021-01-11T11:44:57.056232 #20973] [canary_update(spring-cloud-services/27386766-2ae3-4fd2-b744-ecd3d8666df2 (0))] DEBUG -- DirectorJobRunner: (0.001686s) (conn: 47357401526820) COMMIT
E, [2021-01-11T11:44:57.056833 #20973] [canary_update(spring-cloud-services/27386766-2ae3-4fd2-b744-ecd3d8666df2 (0))] ERROR -- DirectorJobRunner: Error updating canary instance: #<Bosh::Director::RpcRemoteException: Response exceeded maximum allowed length>

The error does not mention space issues because the BOSH Director can't provide a clear message or check for 'sufficient space' since it does not know the implementation of individual BOSH releases. There may be enough space for the bosh-agent to download the package blob, but BOSH can't know how much space a running job will consume.

Resolution

To confirm that you experiencing the same issue, please do the following:

1. ssh into the VM that is being updated using Bosh commands

• bosh -d <deployment name> ssh <vm name>

2. Change to root (sudo -i) and check how much free space the filesystem in the path /var/vcap/data has available.

Sample Output

spring-cloud-services/016bb09b-eea4-4550-aa7a-70743491cc74:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   47M  3.9G   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       2.9G  1.4G  1.4G  50% /
/dev/sdb2       3.9G  1.9G  1.8G  52% /var/vcap/data
tmpfs           1.0M   36K  988K   4% /var/vcap/data/sys/run
/dev/sdc1       9.8G  2.1G  7.2G  23% /var/vcap/store

The amount of space needed will vary but as a rule of thumb ensure that the VM has at least 2 GB of free space in the /var/vcap/data path. If you don't have at least 2 GB available, please increase the size of the VM so that it has enough space for the additional artifacts. 

If the user was to do this on the Spring Cloud Service tile: 

a. Login to the Ops Man UI.
b. Go to the Spring Cloud Services Tile.
c. Go to the Resource Config Tab.
d. Change the VM type to one that has more disk available. 
e. Run an Apply Changes on the SCS tile so that a new VM is created with additional disk space.