NSX Application Platform Upgrade from 4.1.1. to 4.2 fails on NSX Network Detection and Response

Products

• VMware NSX
• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Network Detection and Response
• VMware vDefend Firewall

Issue/Introduction

• Upgrade fails at NSX NDR during an upgrade from NAPP 4.1.1 to NAPP 4.2
  
  
• You will see nsx-metadata-service pod running without any containers. (ssh to NSX manager with root)
  
  root@nsxmgr:~# napp-k get pods | grep metadata
nsx-metadata-service-f5b4cf5df-s2z2h                              0/2     Running     0               5d
nsx-metadata-service-load-feed-cronjob-28795396-ml8k6             1/1     Running     2 (2m33s ago)   10m
• nsx-metadata-service-load-feed-cronjob is also failed or stuck in active. Describing the job will give current status of job.
  
  root@nsxmgr:~# napp-k get jobs | grep metadata
nsx-metadata-service-load-feed-cronjob-28795365            0/1           41m        41m
nsx-metadata-service-load-feed-cronjob-28795396            0/1           11m        11m
• You will see this error in nsx-metadata-service pod and nsx-metadata-service-load-feed-cronjob cronjob logs.
  

  root@nsxmgr:~#napp-k logs job/nsx-metadata-service-load-feed-cronjob-28795396 | grep "The credential is not allowed to access this API"

  2024-09-30 19:24:36,867 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Found NTICS credentials in K8S.
2024-09-30 19:24:36,867 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Checking NTICS credentials...
2024-09-30 19:24:36,867 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - DEBUG - Inizializing NTICS API client...
2024-09-30 19:24:36,868 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Authenticating with NTICS...
2024-09-30 19:24:36,868 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/auth/authenticate
2024-09-30 19:24:37,341 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Authentication successful.
2024-09-30 19:24:37,341 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - NTICS credentials are valid.
2024-09-30 19:24:37,341 - root - INFO - Downloading threat metadata feed from NTICS...
2024-09-30 19:24:37,341 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/auth/authenticate
2024-09-30 19:24:37,754 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/data-feeds/metadata
2024-09-30 19:24:38,013 - ntics_client.handlers - WARNING - API request received permission denied (might retry): status_code: 403; error_details: '{'error_code': '100014', 'error_message': 'The credential is not allowed to access this API'}';
2024-09-30 19:24:38,013 - nsx_metadata_service.load_feed.common - WARNING - Error while connecting to NTICS API server. It might be due to a temporary network or client/server side issue. Retrying one more time... - status_code: 403; error_details: '{'error_code': '100014', 'error_message': 'The credential is not allowed to access this API'}';
2024-09-30 19:25:38,042 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/auth/authenticate
2024-09-30 19:25:38,323 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/data-feeds/metadata

Environment

NSX Application Platform 4.1.1

Cause

This is a known issue affecting NAPP upgrade, when one of the licenses on the NSX manager has expired, which was used to grant the permission to access the threat metadata feed.

Resolution

This issue will be fixed in future release.

Workaround

• Make sure you have Valid License to use NSX NDR and NSX Malware Prevention, otherwise contact your Account Manager.
• Delete the non valid NTICS credentials stored in k8s. This will cause the load feed job to perform another registration with the NTICS service and get new valid credentials. To delete the existing stored credentials you can run the following command:
  
  root@nsxmgr:~# napp-k patch secret nsx-metadata-service-ntics-credentials -p '{"data": {"client_id": null, "client_secret": null}}'
• Verify nsx-metadata-service pod is running and proceed with upgrade.
  
  root@nsxmgr:~# napp-k get pods | grep metadata
nsx-metadata-service-f5b4cf5df-s2z2h                              2/2     Running     0               5d
nsx-metadata-service-load-feed-cronjob-28795396-ml8k6             1/1     Running     2 (2m33s ago)   10m

Additional Information

Current NSX Feature Entitlement
Legacy NSX Feature Entitlement