After upgrade to 10.4.x of Mysql Tile and running errand upgrade all instances some instances might fail
search cancel

After upgrade to 10.4.x of Mysql Tile and running errand upgrade all instances some instances might fail

book

Article ID: 444507

calendar_today

Updated On:

Products

VMware Tanzu Data Services

Issue/Introduction

Running Errand Upgrade-All_instances constantly fails on specific mysql instance

Errand error:

           [upgrade-all-service-instances] 2026/06/16 13:31:14.580545 [upgrade-all] FINISHED PROCESSING Status: FAILED; Summary: Number of successful operations: 1; Number of skipped operations: 0; Number of service instance orphans detected: 1; Number of deleted instances before operation could happen: 0; Number of busy instances which could not be processed: 0; Number of service instances that failed to process: 2 [e7ab9bc3-xxxx-xxxx-xxxx-bd8eaff3a85e, 90519de2-xxxx-xxxx-xxxx-d04cd76d1392]  
           [upgrade-all-service-instances] 2026/06/16 13:31:14.580560 2 errors occurred:  
             
           * [e7ab9bc3-xxxx-xxxx-xxxx-bd8eaff3a85e] Operation failed: bosh task id 14339: Failed for bosh task: 14339  
           * [90519de2-xxxx-xxxx-xxxx-d04cd76d1392] Operation failed: bosh task id 14401: Failed for bosh task: 14401

Bosh Task for the upgrade can have output similar to one below:

bosh task 14339
Using environment '192.168.3.139' as client 'ops_manager'

Task 14339

Task 14339 | 13:05:02 | Preparing deployment: Preparing deployment
....
Task 14339 | 13:05:17 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 14339 | 13:05:18 | Updating instance mysql: mysql/5b5e7d5b-1431-4e9c-b8e7-0075595d11b1 (0) (canary) (00:00:36)
                      L Error: Action Failed get_task: Task 0a7eab79-xxxx-xxxx-xxxx-cf708cef6a17 result: Preparing apply spec: Preparing package mysql-backup-and-recovery-tools: Installing package directory: Decompressing package files: Shelling out to tar: Running command: 'tar --no-same-owner -xf /var/vcap/data/tmp/bosh-http-blob-provider-GET3753777010 -C /var/vcap/data/packages/mysql-backup-and-recovery-tools/cf779f466e9ca043d698bcc986999089b63c34e0', stdout: '', stderr: 'tar: ./bin/restore-server: Wrote only 8192 of 10240 bytes
tar: Exiting with failure status due to previous errors
': exit status 2
Task 14339 | 13:05:54 | Error: Action Failed get_task: Task 0a7eab79-xxxx-xxxx-xxxx-cf708cef6a17 result: Preparing apply spec: Preparing package mysql-backup-and-recovery-tools: Installing package directory: Decompressing package files: Shelling out to tar: Running command: 'tar --no-same-owner -xf /var/vcap/data/tmp/bosh-http-blob-provider-GET3753777010 -C /var/vcap/data/packages/mysql-backup-and-recovery-tools/cf779f466e9ca043d698bcc986999089b63c34e0', stdout: '', stderr: 'tar: ./bin/restore-server: Wrote only 8192 of 10240 bytes
tar: Exiting with failure status due to previous errors
': exit status 2
....
Task 14339 error

Or:

Task 14170 | 10:07:06 | Error: Action Failed get_task: Task 091686be-xxxx-xxxx-xxxx-1afb5925f512 result: Preparing apply spec: Preparing package percona-xtradb-cluster-8.4: Installing package directory: Decompressing package files: Shelling out to tar: Running command: 'tar --no-same-owner -xf /var/vcap/data/tmp/bosh-http-blob-provider-GET3262563709 -C /var/vcap/data/packages/percona-xtradb-cluster-8.4/995066b48aa843bfc5625dee8a58729738493a20', stdout: '', stderr: 'tar: ./bin/mysqlcheck: Wrote only 4608 of 10240 bytes
tar: ./bin/mysql_test_event_tracking: Cannot write: No space left on device
tar: ./LICENSE: Cannot write: No space left on device

Environment

EAR 10.x

VMware Tanzu for MySQL on Tanzu Platform 10.x

Cause

The actual packages size are of about 2 GB and during upgrade process a new version of packages is deployed together with old version 

which double the size of /var/vcap/data to nearly 4 GB, During upgrade process the used space in /var/vcap/data is nearly doubled to 3.7 GB VMs of type medium.mem are on the very limit of available space during the upgrade and can run out os space

/var/vcap/data# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           794M   17M  777M   3% /run
/dev/sda2       4.8G  2.9G  1.8G  63% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda1        48M  5.7M   43M  12% /boot/efi
/dev/sdb2       3.9G  3.7G   17M 100% /var/vcap/data

If the upgrade is complete the space will be reduced back to approximately 2 GB 

In case of disk space error the errand recognise the problem and rolls back and revert the changes to previous version of the mysql, but leaves more space used on the affected VM

bosh -d <SI> vms --vitals

Instance                                            Process State    System      Ephemeral   Persistent
                                                                     Disk Usage  Disk Usage  Disk Usage
mysql-jumpbox/18f83c47-xxxx-xxxx-xxxx-cab752ddb1a5  running          59% (24i%)  17% (0i%)   0% (0i%)
mysql/1a7adfca-xxxx-xxxx-xxxx-d00d0cecdde3          running          59% (24i%)  89% (14i%)  28% (0i%)
mysql/be1658ed-xxxx-xxxx-xxxx-96c7c71bdcd5          running          59% (24i%)  51% (8i%)   28% (0i%)
mysql/dfb4b114-xxxx-xxxx-xxxx-bbcfcf55840f          running          59% (24i%)  51% (8i%)   28% (0i%)

 This issue can be seen on VMs where the ephemeral disk is 8 GB and the size of /var/vcap/data is 4 GB VM types such as meduim.mem

Resolution

The plan have to be updated with another VM type that have more space for ephemeral storage.

Powered by Wolken Software