TAS upgrade fails with updating cloud_controller due to insufficient free space on ephemeral disk

Products

• Operations Manager

Issue/Introduction

When upgrading to TAS 4.x, "Apply Changes" might fail with error as illustrated below when updating cloud_controller instance.

Task 21###076 | 07:42:00 | Updating instance cloud_controller: cloud_controller/50afab58-####-####-####-fac69bf89c3a (0) (canary) (00:00:56)
                         L Error: Action Failed get_task: Task bfca77a6-####-####-####6-ac4da3fa79f4 result: Preparing apply spec: Preparing package ruby-buildpack-cflinuxfs4: Installing package directory: Decompressing package files: Shelling out to tar: Running command: 'tar --no-same-owner -xzf /var/vcap/data/tmp/bosh-http-blob-provider-GET1251279219 -C /var/vcap/data/packages/
ruby-buildpack-cflinuxfs4/1229ee5531405eca03721ce7cd59956e2cc3ee14', stdout: '', stderr: 'tar: ./ruby_buildpack-cached-cflinuxfs4-v1.10.15.zip: Wrote only 9216 of 10240 bytes
tar: Exiting with failure status due to previous errors
': exit status 2

It's also observed high usage of ephemeral disk of the cloud_controller instance in question. The ephemeral disk is 89% used shown by "bosh vms --vitals" command in the following example.

cloud_controller/50afab58-####-####-####-fac69bf89c3a               running        zone-a  #.#.#.# i-yyyy  m5.large   true    bosh-aws-xen-hvm-ubuntu-jammy-go_agent/1.360  Fri Mar  8 19:13:29 UTC 2024  99d 20h 38m 47s  0.27, 0.16, 0.15   -      5.5%   1.4%   2.3%  49% (3.8 GB)  17% (1.3 GB)  57% (24i%)  89% (3i%)   -  

 

Environment

Upgrading TAS to 4.x (for example, 4.0.10 to 4.0.24)

Cause

According to analysis from product team, it's found that during the upgrade process, both the old and new buildpacks are kept on the cloud_controller VM resulting in the upgrade process trying to take way more space than expected and sometimes failure in deploying new buildpack onto the ephemeral disk. 

Resolution

Currently there is no implemented feature to uninstall the unnecessary buildpacks before installing the new ones. The product team are already in the process to create the feature to get it through the entire process and hopefully include it in a future TAS release v6.0. 

Before the new feature is implemented, user can work around this issue by changing the VM type of cloud_controller instance group to some new type with large ephemeral disk. This can be done by choosing a new VM type for cloud_controller instance group on "TAS Settings -> Resource Config" page through Ops Manager web UI, followed by "Apply Changes" against TAS tile.