Upgrading to PKS 1.7 fails during data migration when running clone-db errand for PKS environments created before VMware Enterprise PKS 1.3
search cancel

Upgrading to PKS 1.7 fails during data migration when running clone-db errand for PKS environments created before VMware Enterprise PKS 1.3

book

Article ID: 345545

calendar_today

Updated On:

Products

VMware Cloud PKS

Issue/Introduction

Symptoms:

In VMware Enterprise PKS, the errand “PKS 1.7.x Upgrade - MySQL Clone” is responsible for migrating the PKS control plane data to the PKS DB VM while upgrading to PKS 1.7. This issue is caused by the errand not taking into account leftover tables in the Telemetry database that existed in PKS 1.2 but were removed from new installations in PKS 1.3. These tables have not been forcibly dropped from PKS and exist in PKS installations that have been upgraded from PKS 1.2 to PKS 1.7.

The clone errand attempts to migrate this old Telemetry data but because the PKS 1.7 schema does not have those old tables, the errand fails with the following error message: 

CLONED DATABASE FOR TELEMETRY, BILLING DATABASES FAILED TO LOAD mysqldump FILE  
*Stderr ERROR 1146 (42S02) at line 24: Table 'telemetry.pksdata' doesn't exist  *


Note:

  • This issue does not cause an outage to any clusters. The PKS CLI / API, however, will be unusable until this issue is resolved.
  • This issue does not impact environments where installation was completed using PKS 1.3 and above
  • The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware PKS 1.x

Cause

This is a known issue affecting PKS environments created before VMware Enterprise PKS 1.3 and when upgrading to PKS 1.7.

Resolution

To resolve this issue, drop the old tables from the Telemetry database within the PKS API VM and re-run the “PKS 1.7.x Upgrade - MySQL Clone” errand. After the errand completes successfully, re-run the PKS 1.7 upgrade.

  • If you have not upgraded to VMware Enterprise PKS 1.7 yet

Support recommends following the steps below to proactively prevent this issue from occurring during your upgrade to PKS 1.7. 

  • If you already attempted to upgrade to VMware Enterprise PKS 1.7 and you encounter this issue

Follow the steps below to finalize the PKS 1.7 upgrade. Please do not re-run the PKS 1.7 upgrade before following the steps below. If you have already re-ran the PKS 1.7 upgrade, please follow the steps below and re-run the PKS 1.7 upgrade once more before attempting to resolve any other failures. 



Resolution:

 
  1. Start by SSH'ing into your jumpbox / Ops Manager VM so that you have access to the PKS BOSH deployments.
  2. Run the following command to find the PKS Control Plane deployment by running:
# bosh deployments | grep -i pivotal-container-service

The output  will look something like: 

EX:
"pivotal-container-service-aa1234567bc8de9f0a1c"

  1. Next, run the following command to ssh to the  pivotal-container-service VM
# bosh ssh -d <DEPLOYMENT_NAME pivotal-container-service>
EX:
bosh ssh -d pivotal-container-service-aa1234567bc8de9f0a1c
  1. Now we need to login to the MySQL database that is running on this VM. To locate the password to this database run the following command:
# sudo cat /var/vcap/jobs/pxc-mysql/config/mylogin.cnf
  1.  Copy the username and password from this file.

Ex:

pivotal-container-service/694516dc-5866-4ddc-bc93-201d5bb8ecd8:~$ sudo cat /var/vcap/jobs/pxc-mysql/config/mylogin.cnf             [client]
user="root"
password="yiBLEac7hApcqO3Re-ASAdFL20Y8KTwf"

  1. Run the following command to login to mysql database:
#  mysql -h 127.0.0.1 -P 13306 -u <UserName> -p
Ex:
 mysql -h 127.0.0.1 -P 13306 -u root -p
  1. When promoted enter the password we got from step 5 .
  2. You should see the `mysql >` in the terminal if you logged in successfully. Alternatively, you can use the following command to login to mysql database 
# mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf
  1. Run the following command to switch to the "TELEMETRY" database
#  use telemetry;\
  1. Run the following command to display the tables.
# show tables;

Output Ex;
mysql> show tables;
+---------------------+
| Tables_in_telemetry |
+---------------------+
| pksdata             |
| pkswatermark        |
+---------------------+
  1. If you do not see these tables and have not yet attempted the PKS 1.7 upgrade, then you can safely stop following these steps and continue with the PKS 1.7 upgrade.  You will not see this failure.
 
  1. Run the following command if the tables are present to drop them from the database.
#  drop table pksdata, pkswatermark;

Output ex:
mysql> drop table pksdata, pkswatermark;
Query OK, 0 rows affected (0.08 sec)
  1. Run the following command to verify that tables got dropped.
# show tables;

Ex:
mysql> show tables;
Empty set (0.01 sec)

mysql>
  1. Run the following command to exit out of MySQL database
# \q
  1. Type EXIT  to exit out of the pivotal-container-service VM and return to your jumpbox / Ops Manager VM
  2. If you’re running these steps proactively before initially upgrading to PKS 1.7, you can now stop following these steps and begin the PKS 1.7 upgrade in Ops Manager.
  3.  If you’ve already attempted the PKS 1.7 upgrade and have seen the clone errand fail, run the following command  and the errand should complete successfully. 
# bosh run-errand clone-db
  1. Next, go into Ops Manager and click “Apply Changes” again. Please leave the “PKS 1.7.x Upgrade - MySQL Clone” errand checked. It will confirm the data has been migrated and make sure it will not run the migration again. The upgrade operation to PKS 1.7 should now complete successfully.
​​​​​​​

Additional Information

https://community.pivotal.io/s/article/environments-created-before-PKS-13-upgrading-to-PKS-1-7-fails


Powered by Wolken Software