"pre-start scripts failed. Failed Jobs: policy-server" error Upgrading to CF Networking Release 2.40.0 in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

A migration in CF Networking Release 2.39.0, used in Tanzu Application Service for VMs (TAS for VMs) v2.7.40, v2.9.28, v2.10.20, v2.11.8, introduced a stored procedure that was later dropped via a migration introduced in CF Networking Release 2.40.0 (TAS for VMs v2.7.41+, v2.10.21+, v2.11.9, v2.12.2+).

An upgrade path going from CF Networking Release 2.39.0 to CF Networking Release 2.40.0 may face this error. 

Apply Changes may fail in the following way:

Task 100 | 01:20:17 | L executing pre-start: diego_database/f72272ad-4023-4747-9f17-754b3ab44f9c (0) (canary) (00:10:28)
                        L Error: Action Failed get_task: Task 132d8b8f-f65f-43cb-423d-89e3d6c799f8 result: 1 of 9 pre-start scripts failed. Failed Jobs: policy-server. Successful Jobs: loggregator_agent, sysctl, cfdot, bpm, antivirus, user_add, bosh-dns, syslog_forwarder.

Looking into that diego_database policy-server stdout logs, we see the following error:

diego_database/f72272ad-4023-4747-9f17-754b3ab44f9c.2021-11-24-14-56-30/policy-server/pre-start.stdout.log:{"timestamp":"2021-11-24T01:20:17.154205247Z","level":"error","source":"cfnetworking.policy-server-migrate-db","message":"cfnetworking.policy-server-migrate-db.failed migrating and populating tags, retrying","data":{"error":"perform migrations: executing migration: executor.Exec: Error 1305: PROCEDURE networkpolicyserver.drop_destination_index does not exist handling 66"}}

Environment

Product Version: 2.11

Resolution

Mitigation

To work around this issue, manually add the record to the database.

1. SSH onto the Tanzu Application Service mysql VM and become a sudo user:

mysql/a7576eee-6534-475f-908f-eb7c1984e7cc:~$ sudo su -
mysql/a7576eee-6534-475f-908f-eb7c1984e7cc:~#

2. Connect to mysql client:

mysql/a7576eee-6534-475f-908f-eb7c1984e7cc:~# mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf

3. Manually add the migration 66 entry to the gorp_migrations table:

mysql> insert into networkpolicyserver.gorp_migrations (id, applied_at) values ('66', now());
Query OK, 1 row affected (0.00 sec)

4. Verify that the entry with ID 66 is there:

mysql> select * from networkpolicyserver.gorp_migrations order by applied_at;

Apply Changes should now successfully resume.

Permanent Fix

Upgrade to at least CF Networking Release 2.42.0 (TAS 2.7.43, 2.10.23, 2.11.11, 2.12.4). This version contains a fix to the migration, which will prevent this bug from happening.