Upgrading to Enterprise PKS 1.7 fails during data migration when running clone-db errand for environments created before VMware Enterprise PKS 1.3
search cancel

Upgrading to Enterprise PKS 1.7 fails during data migration when running clone-db errand for environments created before VMware Enterprise PKS 1.3

book

Article ID: 298737

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In Enterprise PKS (now VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)), the errand “PKS 1.7.x Upgrade - MySQL Clone” is responsible for migrating the PKS control plane data to the PKS DB VM while upgrading to Enterprise PKS 1.7. This issue is caused by the errand not taking into account leftover tables in the Telemetry database that existed in Enterprise PKS 1.2 but were removed from new installations in Enterprise PKS 1.3. These tables have not been forcibly dropped from PKS and exist in PKS installations that have been upgraded from PKS 1.2 to PKS 1.7.


The clone errand attempts to migrate this old Telemetry data but because the Enterprise PKS 1.7 schema does not have those old tables, the errand fails with the following error message: 
CLONED DATABASE FOR TELEMETRY, BILLING DATABASES FAILED TO LOAD mysqldump FILE  
*Stderr ERROR 1146 (42S02) at line 24: Table 'telemetry.pksdata' doesn't exist  *

Note

  • This issue does not cause an outage to any clusters. The PKS CLI / API, however, will be unusable until this issue is resolved.
  • This issue does not impact environments where installation was completed using Enterprise PKS 1.3 and above.


Environment

Product Version: Other
OS: Ubuntu

Resolution

To resolve this issue, drop the old tables from the Telemetry database within the PKS API VM and re-run the “PKS 1.7.x Upgrade - MySQL Clone” errand. After the errand completes successfully, re-run the PKS 1.7 upgrade.


If you have not upgraded to VMware Enterprise PKS 1.7

Support recommends following the steps below to proactively prevent this issue from occurring during your upgrade toEnterprise  PKS 1.7. 


If you already attempted to upgrade to VMware Enterprise PKS 1.7 and you encounter this issue

Follow the steps below to finalize the Enterprise PKS 1.7 upgrade. Please do not re-run the Enterprise PKS 1.7 upgrade before following the steps below. If you have already re-ran the Enterprise PKS 1.7 upgrade, please follow the steps below and re-run the Enterprise PKS 1.7 upgrade once more before attempting to resolve any other failures. 


Resolution

1. Start by SSH'ing into your jumpbox or Ops Manager VM so that you have access to the PKS BOSH deployments.

2. Find the PKS Control Plane deployment by running:

bosh deployments | grep “pivotal-container-service

It will look something like: 

“pivotal-container-service-aa1234567bc8de9f0a1c”

3. Next, run bosh ssh -d DEPLOYMENT_NAME pivotal-container-service/0 to ssh into the PKS API VM.

4. Now we need to login to the MySQL database that is running on this VM. To locate the password to this database, open this file with vim: sudo vim /var/vcap/jobs/pxc-mysql/config/mylogin.cnf. Copy the username and password from this file.

5. Login to mysql with: mysql -h 127.0.0.1 -P 13306 -u USERNAME -p PASSWORD. You should see the `mysql >` in the terminal if you logged in successfully. Alternatively, you can use mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf.

6. Run:

use telemetry;\
7. Run:
show tables;
8. Now you see a print out similar to the following:



9. If you do not see these tables and have not yet attempted the Enterprise PKS 1.7 upgrade, then you can safely stop following these steps and continue with your Enterprise PKS 1.7 upgrade. You will not see this failure.

10. Run:
drop table pksdata, pkswatermark;
11. Run show tables; again and verify that mysql returns “Empty set”.

12. Exit out of MySQL and exit out of this VM to return to your jumpbox or Ops Manager VM.

13. If you’re running these steps proactively before initially upgrading to Enterprise PKS 1.7, you can now stop following these steps and begin the Enterprise PKS 1.7 upgrade in Ops Manager.

14. If you’ve already attempted the Enterprise PKS 1.7 upgrade and have seen the clone errand fail, run `bosh run-errand clone-db`. The errand should complete successfully. 

15. Next, go into Ops Manager and click Apply Changes again. Please leave the “PKS 1.7.x Upgrade - MySQL Clone” errand checked. It will confirm the data has been migrated and make sure it will not run the migration again. The upgrade operation to Enterprise PKS 1.7 should now complete successfully.