Upgrading to Enterprise PKS 1.7 fails on the "Running errand PKS 1.7.x Upgrade - MySQL Clone" step
search cancel

Upgrading to Enterprise PKS 1.7 fails on the "Running errand PKS 1.7.x Upgrade - MySQL Clone" step

book

Article ID: 316850

calendar_today

Updated On:

Products

VMware Cloud PKS

Issue/Introduction

Symptoms:
  • Upgrading from Enterprise PKS 1.6.1 to PKS 1.7 fails on "Running errand PKS 1.7.x Upgrade - MySQL Clone) for Enterprise PKS" step.
  • In the install log, you see the entries similar to the following:
VERIFY CHECKSUM FOR UAA AND TELEMETRY Database cloning failed; checksums for UAA and/or Telemetry do not match!

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware PKS 1.x
VMware PKS 1.7.x

Cause

After the PKS-API VM is up, the Mysql Clone DB errand runs. Whenever there is a difference in the entries between the old and new database, the cloning errand fails with the error "VERIFY CHECKSUM FOR UAA AND TELEMETRY Database cloning failed; checksums for UAA and/or Telemetry do not match!" This issue can occur when LDAP is enabled in UAA and  the new database has an updated LDAP entry that the old database does not. 

The clone errand happens after the PKS-API VM is up and between the time UAA comes online after the Enterprise PKS  1.7 upgrade finishes and before the clone-db script finishes cloning. There exists a small window of time where if a record is changed in LDAP (or someone logs in to Enterprise PKS), a record (or records) will change in the new database and cause the clone errand to fail.

 

Resolution

This is a known issue affecting Enterprise PKS 1.7. There is currently no resolution.

Workaround:

When the clone-db errand fails, start troubleshooting the issue by checking the list of entries that are different between the old and new databases. The following steps can be used to compare the two databases:

  1. Login to the pks-db vm from bosh by running:
bosh ssh -d pivotal-container-<service-instance-id> pks-db/<instance-id>
  1. Check the number of entries in the new DB SQL file:
cat /var/vcap/bosh/cloned_mysql_backup.sql  | wc -l
  1. Check the number of entries in the original (old) DB SQL file:
cat /var/vcap/bosh/original_verify_mysql_backup.sql | wc -l
  1. You can run a diff against both files to see what the change is between these two databases or use a utility like IDE tool to compare both the entries.
  2. You can also login to the Mysql UAA DB and then check all the tables and entries in those using the below steps:
 
  1. Log in to PKS API vm by running:

bosh ssh -d DEPLOYMENT_NAME pivotal-container-service/0

  1. Log in to the MySQL database that is running on this VM by running this command:

mysql -h 127.0.0.1 -P 13306 -u USERNAME -p PASSWORD

Note: You can find the username and password information in the /var/vcap/jobs/pxc-mysql/config/mylogin.cnf file.

  1. Access the UAA DB by running use uaa;
  2. To get all the tables and concatenate them, run the following command:

SELECT CONCAT(     'SELECT "',      table_name,      '" AS table_name, COUNT(*) AS exact_row_count FROM '',      table_schema,     ''.'',     table_name,      '' UNION ' )  FROM INFORMATION_SCHEMA.TABLES  WHERE table_schema = 'uaa';

  1. To get the count of all the entries in all the tables in UAA DB, run the following command:

SELECT "authz_approvals" AS table_name, COUNT(*) AS exact_row_count FROM uaa.authz_approvals UNION
SELECT "expiring_code_store" AS table_name, COUNT(*) AS exact_row_count FROM uaa.expiring_code_store UNION  
SELECT "external_group_mapping" AS table_name, COUNT(*) AS exact_row_count FROM uaa.external_group_mapping UNION
SELECT "group_membership" AS table_name, COUNT(*) AS exact_row_count FROM uaa.group_membership UNION
SELECT "groups" AS table_name, COUNT(*) AS exact_row_count FROM uaa.groups UNION
SELECT "identity_provider" AS table_name, COUNT(*) AS exact_row_count FROM uaa.identity_provider UNION
SELECT "identity_zone" AS table_name, COUNT(*) AS exact_row_count FROM uaa.identity_zone UNION
SELECT "mfa_providers" AS table_name, COUNT(*) AS exact_row_count FROM uaa.mfa_providers UNION
SELECT "oauth_client_details" AS table_name, COUNT(*) AS exact_row_count FROM uaa.oauth_client_details UNION
SELECT "oauth_code" AS table_name, COUNT(*) AS exact_row_count FROM uaa.oauth_code UNION
SELECT "revocable_tokens" AS table_name, COUNT(*) AS exact_row_count FROM uaa.revocable_tokens UNION
SELECT "schema_version" AS table_name, COUNT(*) AS exact_row_count FROM uaa.schema_version UNION
SELECT "sec_audit" AS table_name, COUNT(*) AS exact_row_count FROM uaa.sec_audit UNION
SELECT "service_provider" AS table_name, COUNT(*) AS exact_row_count FROM uaa.service_provider UNION
SELECT "user_google_mfa_credentials" AS table_name, COUNT(*) AS exact_row_count FROM uaa.user_google_mfa_credentials UNION
SELECT "user_info" AS table_name, COUNT(*) AS exact_row_count FROM uaa.user_info UNION
SELECT "users" AS table_name, COUNT(*) AS exact_row_count FROM uaa.users ;
 

  1. Log in to the new Database by following the below steps:

bosh ssh -d pivotal-container-<service-instance-id> pks-db/<instance-id>

  1. Log in to the MySQL database that is running on this VM by running the following command:

mysql -h 127.0.0.1 -P 13306 -u USERNAME -p PASSWORD

Note: You can find the username and password information in the mysql --defaults-file available at /var/vcap/jobs/pxc-mysql/config/mylogin.cnf.

  1. Follow the above steps a-c again to get the count of entries in all the tables in the UAA DB and compare them with the old one.

Typically, there will be a mismatch in the user_info table and the new database will have more entries than the old one. You can either insert the missing entry (or entries) into the old UAA database and then run the clone-db errand manually, or you can comment the line that checks the entries between the old and new database and exits the clone-db errand from the "clone-db" script.

  1. To edit the clone-db script login to the pks-db vm using bosh ssh :

bosh ssh -d pivotal-container-<service-instance-id> pks-db/<instance-id>

  1. vi /var/vcap/packages/clone-db/bin/clone-db
  2. Go to line number 201 which has the content "exit 1". Comment the line and then save the file.
  3. Run the clone-db errand manually using a command similar to the following:

bosh -d pivotal-container-<service-instance-id> run-errand clone-db

  1. Once the errand is complete, click the Apply Changes button again in Opsman.


Note: You can perform sanity testing by running the pks clusters and pks get-credentials commands to make sure you are able to list the clusters and login to the clusters. Subsequently, you can run kubectl commands against your clusters to make sure that the kubernetes components are accessible.


Powered by Wolken Software