Upgrade from 3.2.2 to 3.2.3 failing at Pre-upgrade check with NSSgroup UUID Check Null

Products

• VMware NSX

Issue/Introduction

• NSX is Federated
• NSX is upgrading from 3.2.2 to 3.2.3 and the the Pre-Checks are failing at the manager checks with a critical issue: Failed to execute NsServiceGroup UUID check null

• The API GET to list the service groups GET https://<MANAGER IP>/api/v1/ns-service-groups returns a 'null' string at the end of the service entry:

 

• In the NSX-T manager log /var/log/syslog the following text is seen:

2023-06-15T10:04:12.394Z WARN pool-102-thread-3 UpgradeServiceImpl 20409 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=nsServiceGroupUUIDCheck,name=NsServiceGroup UUID check.,description=Check if ID of NSSErviceGroups are incorrect.,componentType=MP] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=nsServiceGroupUUIDCheck,name=NsServiceGroup UUID check.,description=Check if ID of NSSErviceGroups are incorrect.,componentType=MP], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30956,"errorMessage":"Failed to execute NsServiceGroup UUID check.. null"}]}
2023-06-15T10:04:12.394Z INFO pool-102-thread-3 UpgradeServiceImpl 20409 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [PUC] Finish NsServiceGroup UUID check. on component MP. Result -- FAILURE

 

Environment

VMware NSX

Cause

This issue occur due to a realization issue with the service.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

Note: Ensure an up to date backup is available.

• Log in as root to the Local NSX-T manager.
• Run the following grep command to find all the SERVICE_ENTRY and SERVICE which are impacted:

 grep "java.lang.NullPointerException,errorId=PROVIDER_INVOCATION_FAILURE" syslog | awk '{print $14}' | uniq | awk -F ':' '{print $2,$3}' | awk '{print $1,$2}'

Note: If multiple syslogs are gzipped you may need to run the syntax below if you don't get results from above command.

 

zgrep "java.lang.NullPointerException,errorId=PROVIDER_INVOCATION_FAILURE" syslog* | awk '{print $14}' | uniq | awk -F ':' '{print $2,$3}' | awk '{print $1,$2}'

Example result

SVC_TCP_8888,SERVICE_ENTRY TCP/alarms/########-####-####-####-a0a9e4ef8706,
CISCO_DC_Client_svcs,SERVICE_ENTRY CISCO_Kerberos_UDP/alarms/########-####-####-####-a736e5966d87,
SVC_TCP_65023,SERVICE_ENTRY SVC_TCP_65023_TCP1/alarms/########-####-####-####-deb8f4405520,
SVC_UDP_58481,SERVICE_ENTRY UDP/alarms/########-####-####-####-d62ed4bb1006,
CISCO_IVANTIDE_svc,SERVICE_ENTRY CISCO_NETBIOS_SSN/alarms/########-####-####-####-b635b7d7cfdc,
CISCO_IVANTIDE_svc,SERVICE_ENTRY SSH/alarms/########-####-####-####-dc7bc6da6335,
DIS_Active_Directory,SERVICE_ENTRY SOAP/alarms/########-####-####-####-f37b6af5b0f2,
DIS_Active_Directory,SERVICE_ENTRY KERBEROS-UDP/alarms/614fc313-c517-472c-a3c8-39a3e5c5ee99,
CISCO_DC_Client_svcs,SERVICE_ENTRY CISCO_DNS_over_UDP/alarms/########-####-####-####-eabc39141721,
SVC_TCP_51772,SERVICE_ENTRY SVC_TCP_517722/alarms/########-####-####-####-cc9bdc42a753,
SVC_TCP_59639,SERVICE_ENTRY TCP/alarms/########-####-####-####-ce95fea6baf8,
CISCO_IVANTIDE_svc,SERVICE_ENTRY CISCO_NETBIOS_DGM_UDP/alarms/########-####-####-####-f6d15e0f5a91,
SVC_TCP_5048_8168_8268,SERVICE_ENTRY SVC_TCP_5048_8168_8268_TCP1/alarms/########-####-####-####-de

• Log in as root to the global NSX-T manager.
  ​​​Use the following API call's to then update each entry identified from the above grep command.

• First GET each SERVICE_ENTRY and SERVICE identified, replace <service-name> and <service-entry-name> with the one identified:

GET https://<GM-IP>/global-manager/api/v1/global-infra/services/<service-name>/service-entries/<service-entry-name>

Example:
GET https://<GM-IP>/global-manager/api/v1/global-infra/services/SVC_TCP_8888/service-entries/TCP

Output
{
    "protocol_number": 34,
    "resource_type": "IPProtocolServiceEntry",
    "display_name": "TCP",
    "description": "TCP"
}

• Then use the body of the GET API in the following PATCH API

PATCH https://<GM-IP>/global-manager/api/v1/global-infra/services/SVC_TCP_8888/service-entries/TCP

• Sample body:

{
    "protocol_number": 34,
    "resource_type": "IPProtocolServiceEntry",
    "display_name": "TCP",
    "description": "TCP V2"
}

 

Here we have modified the service entry description from TCP to TCP V2.
Note, for the workaround to succeed we need to edit the initial returned payload, so we do this on name and description only, so as not to impact the service.