Enterprise PKS upgrade from 1.3.x to 1.4.0 fails with error: “1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql”

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

• The Enterprise PKS upgrade from 1.3.x to 1.4.0 fails.

• When you run the bosh debug log command, you see output similar to the following:
  
  1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql

• From pxc-mysql pre-start logs, you see messages similar to the following:
  
  panic: Cannot continue, insufficient disk space to complete migration

• When you run the df command, you see that the /var/vcap/store file system is approaching 100% utilization.

• You see several 1GB mysql replication logs under /var/vcap/store/mysql.

Environment

VMware PKS 1.x

Cause

As a part of the Enterprise PKS upgrade, there is a mysql migration from MaridDB(Galera) to MySQL(Percona) which will make a backup for existing files. When /var/vcap/store doesn't have enough free space to handle this requirement, the Enterprise PKS upgrade will fail.

Resolution

This is a known issue affecting Enterprise PKS.

Workaround:

To work around this issue, Increase the size of the PKS VM from Opsmanager -> PKS tile -> Resource from the current configured size to a higher size to have some room for the migration backup.