VMware Aria Operations for Networks integration with VMware NSX fails with "Notification Watcher Webhook Not Registered".

Products

• VCF Operations for Networks
• VMware NSX

Issue/Introduction

• When integrating VMware NSX with Aria Operations for Networks (formerly known as vRealize Network Insight or vRNI) as a data source, an error is encountered stating that the "Notification watcher webhook is not registered to NSX-T."
  
  
  
  
• This error indicates a communication issue between Aria Operations for Networks and the NSX Manager, specifically pertaining to the notification service that uses webhooks.
  
  
• Check the collector logs located at /var/log/arkin/collector/latest.log on the Aria Operations for Networks appliance for the failure.
  
  
• On the NSX Manager in /var/log/phonehome-coordinator/phonehome-coordinator.log, entries similar to the following containing NoSuchMethodError will be visible:

[TIMESTAMP] ERROR http-nio-###.###.###.###-7449-exec-3 NsxBaseRestController 3961 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="monitoring"] Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: com.vmware.nsxapi.nfwconfig.dto.NotificationAuthenticationSchemeDto.getPassword()Lcom/vmware/nsx/management/common/SecureString;
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: com.vmware.nsxapi.nfwconfig.dto.NotificationAuthenticationSchemeDto.getPassword()Lcom/vmware/nsx/management/common/SecureString;
.

.

Caused by: java.lang.NoSuchMethodError: com.vmware.nsxapi.nfwconfig.dto.NotificationAuthenticationSchemeDto.getPassword()Lcom/vmware/nsx/management/common/SecureString;
.

.

[TIMESTAMP] INFO http-nio-###.###.###.###-7449-exec-3 NsxBaseRestController 3961 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="monitoring"] UserName:'admin' ModuleName:'common-services' Operation:'POST@/api/v1/notification-watchers' Operation status: 'failure' Error: General error has occurred.

• Another common symptom is that SVM fails to install when deploying SI (Service Insertion). The following message is logged in location /var/log/proton/nsxapi.log on the NSX Manager.
  
  2024-02-26T04:12:33.591Z  INFO http-nio-###.###.###.###-7440-exec-8 SIRestClient 86180 SERVICE [nsx@6876 comp="nsx-manager" level="INFO" reqId=[REQ UUID] s2comp="service_insertion" subcomp="manager" username=<username>] Rest Request failed due to org.springframework.web.client.HttpClientErrorException$NotFound: 404 : [no body]
  
  
• The previous message alone cannot be used to identify the problem. The previous message means that a HTTP status code 404 without any explanation (i.e. [no body]) is received when installing a SVM. You should use the time stamp of the message and look at NoSuchMethodError message and trace stack above in the above location.

Note: This issue is not impacting any data collection in vRNI / Aria Operation for Networks. All the relevant NSX data (metrics, flows, alerts etc..) are still being collected when the issue happens.

Environment

VMware NSX 4.1.0

VMware NSX 4.1.1

VMware NSX 4.1.2

VMware NSX 4.1.3 

Aria Operations for Networks 6.9

Aria Operations for Networks 6.10

Aria Operations for Networks 6.11

Aria Operations for Networks 6.12

Aria Operations for Networks 6.13

Aria Operations for Networks 6.14

Cause

This known issue is caused by a race condition that occurs during the integration process with NSX. The race condition results in an issue with the Service Manager being instantiated.

Resolution

This issue is resolved in VMware NSX 4.1.2.4
This issue is resolved in VMware NSX 4.2.0

Workaround:
If you are unable to upgrade, contact Broadcom Support for assistance referencing this knowledge article. For more information, see Creating and managing Broadcom support cases.