After upgrading the NCP tile from v2.3.x to v2.4.x new apps cannot be pushed or restaged

Products

• Pivotal CloudFoundry Services 1.x

Issue/Introduction

After upgrading the NSX-T Container Plug-in (NCP) tile from version 2.3.x to version 2.4.x, new apps cannot be pushed or restaged. This a result of the NCP Master election failing.

Note: NCP runs on Pivotal Cloud Foundry (PCF) Diego Database.

The NXS Command Line Interface (NSX CLI) on each Diego Database virtual machine confirms there is no Master. As a result the environment is unstable. The command bellow returns all instances of the Diego Database with no Master status:

From diego database VM:
/var/vcap/jobs/ncp/bin ./nsxcli
get ncp-master status

In NCP 2.3.x and below, High Availability (HA) switching profiles need to be manually created as part of the installation process. 

In NCP 2.4.x, the HA switching profiles will be created automatically. For each cluster a new switching profile is automatically created for Master election.

However if you upgrade from v2.3.x to v2.4.x, it will check for an existing switching profile with the ncp/ha tag and ncp/cluster tag. If a switching profile with the tags is found, it will be modified and a new Master ID tag is added (8ecxxxxxxxxxx ncp/master). This is required for the election process to complete. If you have an older manually added switching profile with the tags, then the Master ID tag is added to this old switching profile.

Therefore the newly created master election profile does not get the Master ID tag added and the election process fails, being stuck in a loop that repeats every 2 minutes.

Resolution

Workaround 1: Before you upgrade

Before upgrading to NCP v2.4.x, you need to delete the old manually created switching profile(s) using the following steps:

In the NSX-T UI:

1. Go to: Networking / Switching / Switching Profiles
2. Select ncp-ha (for all clusters)
3. Delete this switching profile
4. Once done, proceed with the upgrade.

Workaround 2: After you upgrade

If you have already upgraded to NCP 2.4.x and follow these steps:

In the NSX-T user interface (UI):

1. Go to: Networking > Switching > Switching Profiles
2. Select ncp-ha for all clusters.
3. Delete this switching profile.
4. A new switching profile will be created automatically in few minutes or an existing Master election switching profile will be modified and the environment should go back to being stable with Master node elected.

Note: This issue does not impact new NCP 2.4 deployments. This issue impacts upgrading from NCP 2.4+ to NCP 2.4.x. This workaround must be applied before you upgrade.