Central Control Plane fails to reconnect with Corfu table and does not process entity updates on NSX-T 3.2.0 or 3.2.0.1

Products

• VMware NSX

Issue/Introduction

Symptoms:
Entity updates are not being processed by the Central Control Plane on NSX-T 3.2.0 or 3.2.0.1. Changes made on the management plane may not become effective.
 
/var/log/cloudnet/nsx-ccp* logs on NSX Manager show "Failed to re-subscribe" errors:
 
2022-03-17T23:22:03.477Z ERROR org.corfudb.runtime.collections.streaming.StreamPollingScheduler-worker-1 ResumeStreamListener 10453 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP4" level="ERROR" subcomp="ccp"] Failed to re-subscribe [tag:ccp] nsx$[null]. Listener is NOT SUBSCRIBED yet! lastProcessedTs:epoch: 315
sequence: 1730001111
, retry 1/20
 
and after 20 failures to resubscribe:

2022-03-17T23:22:36.423Z ERROR org.corfudb.runtime.collections.streaming.StreamPollingScheduler-worker-1 ResumeStreamListener 10453 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP4" level="ERROR" subcomp="ccp"] Failed to re-subscribe [tag:ccp] nsx$[null]. Listener is NOT SUBSCRIBED! lastProcessedTs:epoch: 315
sequence: 1730001111
. Last retry failed 20/20 ENDING!

Cause

After a temporary disconnect with Corfu, the Central Control Plane fails to re-subscribe to the UFO table. CCP can no longer read UFO updates from Corfu, and new entity updates are no longer processed by the Central Control Plane.

Resolution

This issue is resolved in NSX-T versions 3.2.1 and 4.0.0

Workaround:
Restart Controller service from the Manager root shell on all 3 Manager nodes one at a time:
root@manager# /etc/init.d/nsx-ccp restart