NSX Service Insertion creation for 3rd party security appliance failed

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• In NSX 4.x version before 4.1.2.4, when trying to deploy a security appliance that utilize NSX service insertion, the creation of such service could fail. 
• This may also happen when you already have a Service Insertion deployed but later deleted and the issue raise during recreation of the same or new service insertion. 
• You may see entries similar to the following in /var/log/proton/nsxapi.log



  nsxapi.10.log:1244185:2024-03-11T14:01:53.990Z ERROR http-nio-127.0.0.1-7440-exec-15 SINotificationServiceImpl 5124 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42735" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="admin"] Service Manager Update operation failed with exception 404 : [no body].
  
  nsxapi.10.log:1244186:2024-03-11T14:01:53.992Z  INFO http-nio-127.0.0.1-7440-exec-15 NsxBaseRestController 5124 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/serviceinsertion/service-managers/<service-manager-UUID> caused by exception com.vmware.nsx.management.serviceinsertion.exception.ServiceInsertionException:  {"moduleName":"SERVICE","errorCode":42735,"errorMessage":"Service Manager Update operation failed."}
  
  2024-03-11T14:01:53.986Z ERROR http-nio-127.0.0.1-7449-exec-2 NsxBaseRestController 4031 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="monitoring"] Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: com.vmware.nsxapi.nfwconfig.dto.NotificationAuthenticationSchemeDto.getPassword()Lcom/vmware/nsx/management/common/SecureString;
  org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: 
  
  2023-09-26T07:48:05.071Z ERROR http-nio-127.0.0.1-7440-exec-21 SINotificationServiceImpl 4356 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42735" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="admin"] Service Manager Creation operation failed with exception 404 : [no body].
  2023-09-26T07:48:05.073Z  INFO http-nio-127.0.0.1-7440-exec-21 NsxBaseRestController 4356 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/serviceinsertion/service-managers caused by exception com.vmware.nsx.management.serviceinsertion.exception.ServiceInsertionException:  {"moduleName":"SERVICE","errorCode":42735,"errorMessage":"Service Manager Creation operation failed."}
  
  2023-09-26T07:50:00.792Z ERROR http-nio-127.0.0.1-7440-exec-16 ServiceInsertionFacadeValidatorImpl 4356 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42231" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="admin"] Service definition display name <Service Name> has been used already.
  
  2024-06-25T18:10:07.356Z ERROR http-nio-127.0.0.1-7440-exec-42 SINotificationServiceImpl 85987 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42735" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="<user-ID>"] Service Manager Creation operation failed with exception 400 : "{<EOL>  "httpStatus" : "BAD_REQUEST",<EOL>  "error_code" : 301120,<EOL>  "module_name" : "NotificationFramework",<EOL>  "error_message" : "Watcher with specified IP and port configuration already exists, see /api/v1/notification-watchers/<UUID>"<EOL>}".
  2024-06-25T18:10:07.358Z  INFO http-nio-127.0.0.1-7440-exec-42 NsxBaseRestController 85987 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/serviceinsertion/service-managers/ caused by exception com.vmware.nsx.management.serviceinsertion.exception.ServiceInsertionException:  {"moduleName":"SERVICE","errorCode":42735,"errorMessage":"Service Manager Creation operation failed."}
  2024-06-25T18:10:07.358Z  INFO http-nio-127.0.0.1-7440-exec-42 NsxBaseRestController 85987 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'<user-ID>' ModuleName:'SERVICE' Operation:'POST@/api/v1/serviceinsertion/service-managers/' Operation status: 'failure' Error: Service Manager Creation operation failed.
  2024-06-25T18:10:07.356Z ERROR http-nio-127.0.0.1-7440-exec-42 SINotificationServiceImpl 85987 SERVICE [nsx@6876 comp="nsx-manager" errorCode="MP42735" level="ERROR" reqId="<request-id>" s2comp="service_insertion" subcomp="manager" username="<user-ID>"] Service Manager Creation operation failed with exception 400 : "{<EOL>  "httpStatus" : "BAD_REQUEST",<EOL>  "error_code" : 301120,<EOL>  "module_name" : "NotificationFramework",<EOL>  "error_message" : "Watcher with specified IP and port configuration already exists, see /api/v1/notification-watchers/<UUID>"<EOL>}".

Environment

VMware NSX (versions prior to 4.1.2.4)

Resolution

This issue is resolved in VMware NSX 4.1.2.4
This issue is resolved in VMware NSX 4.2.0

Workaround:

Contact Broadcom Support and reference this KB.