nsx-metadata-service
pod running without any containers. (ssh to NSX manager with root)root@nsxmgr:~# napp-k get pods | grep metadata
nsx-metadata-service-f5b4cf5df-s2z2h 0/2 Running 0 5d
nsx-metadata-service-load-feed-cronjob-28795396-ml8k6 1/1 Running 2 (2m33s ago) 10m
nsx-metadata-service-load-feed-cronjob
is also failed or stuck in active. Describing the job will give current status of job.root@nsxmgr:~# napp-k get jobs | grep metadata
nsx-metadata-service-load-feed-cronjob-28795365 0/1 41m 41m
nsx-metadata-service-load-feed-cronjob-28795396 0/1 11m 11m
nsx-metadata-service
pod and nsx-metadata-service-load-feed-cronjob
cronjob logs.root@nsxmgr:~#napp-k logs job/nsx-metadata-service-load-feed-cronjob-28795396
| grep "The credential is not allowed to access this API"
2024-09-30 19:24:36,867 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Found NTICS credentials in K8S.
2024-09-30 19:24:36,867 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Checking NTICS credentials...
2024-09-30 19:24:36,867 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - DEBUG - Inizializing NTICS API client...
2024-09-30 19:24:36,868 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Authenticating with NTICS...
2024-09-30 19:24:36,868 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/auth/authenticate
2024-09-30 19:24:37,341 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - Authentication successful.
2024-09-30 19:24:37,341 - nsx_metadata_service.load_feed.ntics_credentials_on_premise - INFO - NTICS credentials are valid.
2024-09-30 19:24:37,341 - root - INFO - Downloading threat metadata feed from NTICS...
2024-09-30 19:24:37,341 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/auth/authenticate
2024-09-30 19:24:37,754 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/data-feeds/metadata
2024-09-30 19:24:38,013 - ntics_client.handlers - WARNING - API request received permission denied (might retry): status_code: 403; error_details: '{'error_code': '100014', 'error_message': 'The credential is not allowed to access this API'}';
2024-09-30 19:24:38,013 - nsx_metadata_service.load_feed.common - WARNING - Error while connecting to NTICS API server. It might be due to a temporary network or client/server side issue. Retrying one more time... - status_code: 403; error_details: '{'error_code': '100014', 'error_message': 'The credential is not allowed to access this API'}';
2024-09-30 19:25:38,042 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/auth/authenticate
2024-09-30 19:25:38,323 - ntics_client.handlers - DEBUG - requesting https://api.prod.nsxti.vmware.com/1.0/data-feeds/metadata
NSX Application Platform 4.1.1
This is a known issue affecting NAPP upgrade, when one of the licenses on the NSX manager has expired, which was used to grant the permission to access the threat metadata feed.
This issue will be fixed in future release.
Workaround
root@nsxmgr:~# napp-k patch secret nsx-metadata-service-ntics-credentials -p '{"data": {"client_id": null, "client_secret": null}}'
nsx-metadata-service
pod is running and proceed with upgrade.root@nsxmgr:~# napp-k get pods | grep metadata
nsx-metadata-service-f5b4cf5df-s2z2h 2/2 Running 0 5d
nsx-metadata-service-load-feed-cronjob-28795396-ml8k6 1/1 Running 2 (2m33s ago) 10m