This issue is resolved in NCP 3.1.2 which permanently sets wait_for_security_policy_sync to False.
Workaround:
If the Controller is already down, please open a Support Request and the stale DB entries can be removed by a support engineer.
To prevent the issue from occurring, please apply the following configuration change.
For Openshift environments:1) Gain admin access to the Openshift cluster from a console.
2) Ensure the operator is running. This can be done in two ways:
A) oc get pods -n nsx-system-operator -> should return a running pod for nsx-ncp-operator
B) oc get co nsx-ncp -> should return the operator as "available"
3) Edit the operator configmap
oc edit cm -n nsx-system-operator nsx-ncp-operator-config
4) Find the [nsx_v3] section and set wait_for_security_policy_sync = False (this line will need to be added)
5) Save the config map (:wq)
6) Wait for the operator to update the config map and recreate the ncp pod
oc get pods -n nsx-system
7) Confirm the config update went correctly
oc get cm -n nsx-system nsx-ncp-config -o yaml
It should now contain: wait_for_security_policy_sync = False
8) Agent pods should be confirmed to be in a "Running" state and to have been recently restarted.
For Vanilla Kubernetes environments:1)
kubectl edit cm -n nsx-system nsx-ncp-config -> set flag to False
2) delete NCP pods (There is no need to restart agents)