Enterprise PKS upgrade from 1.3.x to 1.4.0 fails with error: “1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql”
search cancel

Enterprise PKS upgrade from 1.3.x to 1.4.0 fails with error: “1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql”

book

Article ID: 316778

calendar_today

Updated On:

Products

VMware Cloud PKS

Issue/Introduction

Symptoms:
  • The Enterprise PKS upgrade from 1.3.x to 1.4.0 fails.

  • When you run the bosh debug log command, you see output similar to the following:

    1 of 8 pre-start scripts failed. Failed Jobs:pxc-mysql

  • From pxc-mysql pre-start logs, you see messages similar to the following:

    panic: Cannot continue, insufficient disk space to complete migration

  • When you run the df command, you see that the /var/vcap/store file system is approaching 100% utilization.

  • You see several 1GB mysql replication logs under /var/vcap/store/mysql.



Environment

VMware PKS 1.x

Cause

As a part of the Enterprise PKS upgrade, there is a mysql migration from MaridDB(Galera) to MySQL(Percona) which will make a backup for existing files. When /var/vcap/store doesn't have enough free space to handle this requirement, the Enterprise PKS upgrade will fail.

Resolution

This is a known issue affecting Enterprise PKS.

Workaround:

To work around this issue, Increase the size of the PKS VM from Opsmanager -> PKS tile -> Resource from the current configured size to a higher size to have some room for the migration backup.