Service Manager Creation operation failed

Products

• VMware vDefend Firewall

Issue/Introduction

Issue:In NSX 3.2.3.1 when trying to deploy a SI error "Service Manager Creation operation failed" has generated.

Following logs seen in syslog and api.log

/var/log/proton/syslog
2024-06-19T05:15:12.644Z <nsxt-manager> NSX 50800 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="monitoring"] Handler dispatch failed; nested exception is java.lang.NoSuchMethodError:
com.vmware.nsxapi.nfwconfig.dto.NotificationAuthenticationSchemeDto.getPassword()Lcom/vmware/nsx/management/common/SecureString;

2024-06-19T05:15:12.647Z <nsxt-manager> NSX 50800 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="monitoring"] UserName:'admin' ModuleName:'common-services' Operation:'POST@/api/v1/notification-watchers/' Operation status: 'failure' Error: General error has occurred.

2024-06-19T05:15:12.679Z <nsxt-manager> NSX 53273 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42735" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="admin"] Service Manager Creation operation failed with exception 404 : [no body].

/var/log/proton/nsxapi.log
2024-04-19T04:52:26.326Z <nsx-manager> NSX 3480 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="monitoring"] UserNa
me:'admin' ModuleName:'NotificationFramework' Operation:'POST@/api/v1/notification-watchers/' Operation status: 'failure' Error: Watcher with specified IP and port configuration already exists, see /api/v1/notification-watchers/<UUID>

2024-04-19T04:52:26.327Z <nsx-manager> NSX 4431 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42735" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="admin"] Service Manager Creation operation failed with exception 400 : "{<EOL>  "httpStatus" : "BAD_REQUEST",<EOL>  "error_code" : 301120,<EOL>  "module_name" : "NotificationFramework",<EOL>  "error_message" : "Watcher with specified IP and port configuration already exists, see /api/v1/notification-watchers/<UUID>"<EOL>}".

Environment

NSX-T 3.2.3.1

Cause

This issue occurs when stale notification-watcher IPs are present in the environment.

Resolution

Check if following table have stale entry

/opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t NfwNotificationWatcherConfig >NfwNotificationWatcherConfig.txt

If you see stale entry in NfwNotificationWatcherConfig then raise case with Broadcom support to clean up stale entry