After restarting the collector node, a message stating, "Notification watcher webhook not registered to NSX-T" is present for the NSX-T data source

Products

• VCF Operations for Networks
• VMware NSX

Issue/Introduction

• VCF Operations for Networks 9.x is configured with a VMware NSX-T Manager data source.
• After a restart of the collector node, the following message is displayed over the VMware NSX-T Manager data source:
  
  Notification watcher webhook not registered to NSX-T

• You see messages similar to the following in the /var/log/arkin/collector.log on the collector node:
  
  

  [Timestamp] WARNING collector 9853 [netw@4413 class="common.utils.CommonUtils" thread="NSXT_######_Config_OpMgr-5" method="logException" line="2743"] Notification watcher webhook not registered in NSX-T.com.vnera.dataproviders.core.impl.vmware.nsx.utils.StateWatcherException: Notification watcher webhook not registered in NSX-T. 

  [Timestamp] ERROR collector 9853 [netw@ class="dataprovider.utils.HttpUtils" thread="Task_NSXT_######-0" method="checkStatusAndThrow" line="41"] API /api/v1/notification-watchers error response {
    "httpStatus" : "BAD_REQUEST",
    "error_code" : 301129,
    "module_name" : "NotificationFramework",
    "error_message" : "A previous watcher using the same server and port is in the process of being removed. Please retry after waiting 10 seconds."
  }
  

• When you query the NSX API endpoint, https://<NSX Manager IP/FQDN>/api/v1/notification-watchers, you do not see any output that references the IP address of the collector node.

Environment

• VCF Operations for Networks 9.x
• VMware NSX 9.0.0

Cause

When the VCF Operations for Networks collector node is restarted, the webhook configuration created in NSX will be deleted and recreated. In NSX 9.0.0, there is a 10 minute minimum time frame between when the webhook configuration was last created and when it can be recreated. If the collector node is restarted within 10 minutes of the last time the webhook configuration was recreated, the webhook configuration will be deleted but not recreated.

Resolution

This issue would be resolved in future version for VCF 9.X

Workaround :

• Delete the VMware NSX-T Manager data source in VCF Operations GUI
• Wait for 24 hours
• Add the VMware NSX-T Manager data source in VCF Operations GUI

 

To add datasource, see Configuring the NSX Account in VCF Operations

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

This issue could also occur in VMware NSX 9.0.0 environments where the webhook configuration is removed shortly after being created, outside of a restart of the collector node.

If you are contacting Broadcom support about this issue, please provide the following:

• NSX and VCF Operations for Networks support bundles (NSX manager nodes only, VCF Operations for Networks platform and collector nodes)
• The output from the following API call against NSX:
  
  GET https://<NSX Manager IP/FQDN>/api/v1/notification-watchers