After the PKS-API VM is up, the Mysql Clone DB errand runs. Whenever there is a difference in the entries between the old and new database, the cloning errand fails with the error "VERIFY CHECKSUM FOR UAA AND TELEMETRY Database cloning failed; checksums for UAA and/or Telemetry do not match!" This issue can occur when LDAP is enabled in UAA and the new database has an updated LDAP entry that the old database does not.
The clone errand happens after the PKS-API VM is up and between the time UAA comes online after the Enterprise PKS 1.7 upgrade finishes and before the clone-db script finishes cloning. There exists a small window of time where if a record is changed in LDAP (or someone logs in to Enterprise PKS), a record (or records) will change in the new database and cause the clone errand to fail.
When the clone-db errand fails, start troubleshooting the issue by checking the list of entries that are different between the old and new databases. The following steps can be used to compare the two databases:
bosh ssh -d DEPLOYMENT_NAME pivotal-container-service/0
mysql -h 127.0.0.1 -P 13306 -u USERNAME -p PASSWORD
Note: You can find the username and password information in the /var/vcap/jobs/pxc-mysql/config/mylogin.cnf file.
SELECT CONCAT( 'SELECT "', table_name, '" AS table_name, COUNT(*) AS exact_row_count FROM '', table_schema, ''.'', table_name, '' UNION ' ) FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = 'uaa';
SELECT "authz_approvals" AS table_name, COUNT(*) AS exact_row_count FROM uaa.authz_approvals UNION
SELECT "expiring_code_store" AS table_name, COUNT(*) AS exact_row_count FROM uaa.expiring_code_store UNION
SELECT "external_group_mapping" AS table_name, COUNT(*) AS exact_row_count FROM uaa.external_group_mapping UNION
SELECT "group_membership" AS table_name, COUNT(*) AS exact_row_count FROM uaa.group_membership UNION
SELECT "groups" AS table_name, COUNT(*) AS exact_row_count FROM uaa.groups UNION
SELECT "identity_provider" AS table_name, COUNT(*) AS exact_row_count FROM uaa.identity_provider UNION
SELECT "identity_zone" AS table_name, COUNT(*) AS exact_row_count FROM uaa.identity_zone UNION
SELECT "mfa_providers" AS table_name, COUNT(*) AS exact_row_count FROM uaa.mfa_providers UNION
SELECT "oauth_client_details" AS table_name, COUNT(*) AS exact_row_count FROM uaa.oauth_client_details UNION
SELECT "oauth_code" AS table_name, COUNT(*) AS exact_row_count FROM uaa.oauth_code UNION
SELECT "revocable_tokens" AS table_name, COUNT(*) AS exact_row_count FROM uaa.revocable_tokens UNION
SELECT "schema_version" AS table_name, COUNT(*) AS exact_row_count FROM uaa.schema_version UNION
SELECT "sec_audit" AS table_name, COUNT(*) AS exact_row_count FROM uaa.sec_audit UNION
SELECT "service_provider" AS table_name, COUNT(*) AS exact_row_count FROM uaa.service_provider UNION
SELECT "user_google_mfa_credentials" AS table_name, COUNT(*) AS exact_row_count FROM uaa.user_google_mfa_credentials UNION
SELECT "user_info" AS table_name, COUNT(*) AS exact_row_count FROM uaa.user_info UNION
SELECT "users" AS table_name, COUNT(*) AS exact_row_count FROM uaa.users ;
bosh ssh -d pivotal-container-<service-instance-id> pks-db/<instance-id>
mysql -h 127.0.0.1 -P 13306 -u USERNAME -p PASSWORD
Note: You can find the username and password information in the mysql --defaults-file available at /var/vcap/jobs/pxc-mysql/config/mylogin.cnf.
Typically, there will be a mismatch in the user_info table and the new database will have more entries than the old one. You can either insert the missing entry (or entries) into the old UAA database and then run the clone-db errand manually, or you can comment the line that checks the entries between the old and new database and exits the clone-db errand from the "clone-db" script.
bosh ssh -d pivotal-container-<service-instance-id> pks-db/<instance-id>
bosh -d pivotal-container-<service-instance-id> run-errand clone-db
Note: You can perform sanity testing by running the pks clusters and pks get-credentials commands to make sure you are able to list the clusters and login to the clusters. Subsequently, you can run kubectl commands against your clusters to make sure that the kubernetes components are accessible.