Upgrade of tile VMs fails with "Response exceeded maximum allowed length"

Products

• Support Only for Spring

Issue/Introduction

During the upgrade of a tile, for example, the Spring Cloud Services, new packages, logs, and releases need to be added to the VM under the /var/vcap/data path. 

If there is not enough space to allocate these new resources the deployment or Apply Changes fails with the following error: 

task 983507 | 11:44:43 | Updating instance spring-cloud-services: spring-cloud-services/27386766-2ae3-4fd2-b744-ecd3d8666df2 (0) (canary) (00:00:14)
Updating deployment:
Expected task '983507' to succeed but state is 'error'
Exit code 1
L Error: Response exceeded maximum allowed length

The Debug logs also show RpcRemoteException errors:

D, [2021-01-11T11:44:57.056232 #20973] [canary_update(spring-cloud-services/27386766-2ae3-4fd2-b744-ecd3d8666df2 (0))] DEBUG -- DirectorJobRunner: (0.001686s) (conn: 47357401526820) COMMIT
E, [2021-01-11T11:44:57.056833 #20973] [canary_update(spring-cloud-services/27386766-2ae3-4fd2-b744-ecd3d8666df2 (0))] ERROR -- DirectorJobRunner: Error updating canary instance: #<Bosh::Director::RpcRemoteException: Response exceeded maximum allowed length>

The error does not mention space issues because the BOSH Director can't provide a clear message or check for 'sufficient space' since it does not know the implementation of individual BOSH releases. There may be enough space for the bosh-agent to download the package blob, but BOSH can't know how much space a running job will consume.

Environment

Tanzu Platform  
Operations Manager 
Spring Cloud Config Services/Application Services 

Cause

Low Ephemeral disk 

Resolution

To confirm that you experiencing the same issue, please do the following:

1. ssh into the VM that is being updated using Bosh commands

• bosh -d <deployment name> ssh <vm name>

2. Change to root (sudo -i) and check how much free space the filesystem in the path /var/vcap/data has available.

Sample Output

spring-cloud-services/016bb09b-eea4-4550-aa7a-70743491cc74:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   47M  3.9G   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       2.9G  1.4G  1.4G  50% /
/dev/sdb2       3.9G  1.9G  1.8G  52% /var/vcap/data
tmpfs           1.0M   36K  988K   4% /var/vcap/data/sys/run
/dev/sdc1       9.8G  2.1G  7.2G  23% /var/vcap/store

The amount of space needed will vary but as a rule of thumb ensure that you have more than 50% of free space in /var/vcap/data path. If you don't have that available, please increase the size of the VM so that it has enough space for the additional artifacts. 

If the user was to do this on the Spring Cloud Service tile: 

a. Login to the Ops Man UI.
b. Go to the Spring Cloud Services Tile.
c. Go to the Resource Config Tab.
d. Change the VM type to one that has more disk available. 
e. Run an Apply Changes on the SCS tile so that a new VM is created with additional disk space. 

Additional Information

Apply the same solution if you face this issue in diego_database VM

Task 494566 | 05:44:59 | Updating instance diego_database: diego_database/2e65433b-c567-g6d7-9t67-01d4568c0u3d (1) (canary) (00:00:14)

Updating deployment:  
    Expected task '494566' to succeed but state is 'error'                        
                                 L Error: Response exceeded maximum allowed length

Task 494566 | 05:45:13 | Error: Response exceeded maximum allowed length
Task 494566 Started  Sat May 15 06:36:43 UTC 2025
Task 494566 Finished Sat May 15 07:47:33 UTC 2025
Task 494566 Duration 00:11:20
Task 494566 error  
Exit code 1