Error: "The requested resource [/api/2.0/global/config] is not available" in HCX Manager logs after reboot of services

Products

• VMware HCX

Issue/Introduction

After a fresh installation of VMware HCX, or after a reboot or restart of the HCX application services, an ERROR entry appears in the HCX Manager application log. HCX sends a request to the registered NSX Manager and NSX rejects the request as unavailable.

The full error text in /common/logs/admin/app.log is:

{"module_name":"common-service","error_message":"The requested resource [/api/2.0/global/config] is not available","error_code":98}

The error is immediately followed by a failed job and a stack trace:

<date>T19:36:02.370Z INFO hcx-app-engine [logger="c.v.v.h.a.n.r.c.ApplianceManagementControllerProxy" thread="NSXService_SvcThread-7"] Getting VCenter associated with Nsx: https://<nsx-manager-fqdn>/api/2.0/global/config
<date>T19:36:02.443Z INFO hcx-app-engine [logger="c.v.v.h.a.n.r.RestTemplateFactory$PreemptiveAuthInterceptor"] Successfully added preemptive Authorization header to request: /api/2.0/global/config (username: svc-hcx-nsx-####)
<date>T19:36:02.453Z ERROR hcx-app-engine [logger="c.v.v.h.a.n.e.NetworkSecurityErrorHandler"] Response error xml : {"module_name":"common-service","error_message":"The requested resource [/api/2.0/global/config] is not available","error_code":98}
<date>T19:36:02.455Z ERROR hcx-app-engine [logger="c.v.v.h.s.n.VimNsxMappingNsxSubworkflowJob"] VimNsxMappingPopulationJob Nsx for failed
org.springframework.web.client.RestClientException: Not Found
at com.vmware.vchs.hybridity.adapters.nsx.error.NetworkSecurityErrorHandler.handleClientOrServerError(NetworkSecurityErrorHandler.java:130)
at com.vmware.vchs.hybridity.adapters.nsx.rest.controllers.ApplianceManagementControllerProxy.getVcNsxMappingInfo(ApplianceManagementControllerProxy.java:43)
at com.vmware.vchs.hybridity.service.nsxservice.VimNsxMappingNsxSubworkflowJob.getvCenterInstanceUuid(VimNsxMappingNsxSubworkflowJob.java:101)
at com.vmware.vchs.hybridity.service.nsxservice.VimNsxMappingNsxSubworkflowJob.run(VimNsxMappingNsxSubworkflowJob.java:48)

This article applies when all of the following are true:

• The error appears only shortly after the HCX application services start, and does not repeat between restarts.
• The failing request path is /api/2.0/global/config.
• No alert or error is raised in the HCX user interface.
• NSX shows a healthy status in the HCX appliance management interface.
• Service Mesh, Network Extension, and migration operations continue to function normally.

Environment

• VMware HCX Manager
• VMware NSX 4.x and later

Cause

HCX sends the request using a legacy API path format that current NSX versions no longer serve. NSX 4.x and later publish only /api/v1/ (manager) and /policy/api/v1/ (policy) endpoints. The path /api/2.0/global/config is not a published endpoint in the current NSX API, so NSX correctly answers with HTTP 404 and error code 98.

The request originates from ApplianceManagementControllerProxy.getVcNsxMappingInfo(), called by VimNsxMappingNsxSubworkflowJob.getvCenterInstanceUuid(). The purpose of that subworkflow is to retrieve the vCenter Server instance UUID associated with NSX.

That value is already stored by HCX during NSX registration, and all NSX inventory data used by Network Profiles, Compute Profiles, and Service Mesh is retrieved successfully through the supported /api/v1/ and /policy/api/v1/ endpoints. The legacy path remains present in the HCX code but is redundant against NSX 4.x and later, so its failure has no functional effect.

Resolution

No action is required. This log entry is expected when HCX is registered to NSX 4.x and later, and can be disregarded. It appears only at HCX application service startup, does not retry, and does not raise an alert.

Do not modify the NSX registration, the NSX service account, or any HCX configuration in response to this message. The legacy path resides in the HCX program code, so no configuration change removes it.

Use the steps below to confirm that the environment matches this article and that no HCX function is impaired.

Log locations

• HCX Manager
  • /common/logs/admin/app.log - application engine and NSX job activity
  • /common/logs/appliance-management/appliance-management.log - NSX registration record
• NSX Manager
  • /var/log/proxy/envoy_access_log.txt* - inbound API requests and response codes
  • /var/log/nsx-audit.log* - audited API operations and outcomes
  • /var/log/nvpapi/api_access.log* - node management API access

The same paths apply whether the logs are read directly over SSH on the appliance, or from an extracted HCX or NSX support bundle. When using a support bundle, prefix each path with the extracted bundle directory.

Step 1 - Confirm the error and how often it occurs

Run on the HCX Manager, or against an extracted HCX support bundle:

grep -n "api/2.0/global/config" /common/logs/admin/app.log

Expected result: a small number of entries, each occurring within roughly 90 seconds of an HCX application service start. Entries that repeat continuously, or that appear outside of service startup, indicate a different issue.

Step 2 - Confirm the NSX registration is healthy

grep "ServiceConfigDAO" /common/logs/appliance-management/appliance-management.log | grep -i nsx

Expected result: a registration record containing the fields below. The populated vcuuid is the same value that the failing request attempts to retrieve, which confirms HCX already holds it.

Upserted item: {"config":{"userName":"svc-hcx-nsx-####","url":"https://<nsx-manager-fqdn>","type":"NSX-T","nsxuuid":"<nsx-node-uuid>","auth":{"authType":"BASIC"},"vcuuid":"<vcenter-instance-uuid>","nodeType":"LM","isFederated":false,"version":"<version>"}, ... "status":"JOINED" ... }, section: nsx

Step 3 - Confirm the supported NSX API calls succeed

grep "NSXService_SvcThread" /common/logs/admin/app.log | grep -o 'logger="[^"]*"' | sort | uniq -c | sort -rn

Expected result: the NSX inventory jobs dominate the output, and the legacy proxy classes appear only twice each.

72 logger="c.v.v.h.s.n.NsxtVPCSubnetSyncJob"
26 logger="c.v.v.h.s.n.NSXInventoryService"
24 logger="c.v.v.h.s.n.NsxTInventoryCleanupJob"
12 logger="c.v.v.h.s.n.NsxTInventorySyncJob"
2 logger="c.v.v.h.a.n.r.c.ApplianceManagementControllerProxy"
2 logger="c.v.v.h.a.n.e.NetworkSecurityErrorHandler"

Confirm the inventory jobs reach a completed state and continue on their normal cycle:

grep -c "NsxtVPCSubnetSyncJob.*COMPLETED" /common/logs/admin/app.log
grep "NsxTInventorySyncJob" /common/logs/admin/app.log | tail

Expected result: a non-zero count of completions, and NsxTInventorySyncJob entries repeating on an approximate five minute interval up to the point of log collection.

Step 4 - Confirm HCX classifies the failure as non-reporting

grep "from notifying" /common/logs/admin/app.log

Expected result includes the following line, which shows that HCX suppresses this job type from raising an administrator alert:

Skipping the jobType NSXServiceJob and workflow Type VimNsxMappingNsxSubworkflow from notifying.

Step 5 - Confirm the behaviour from the NSX side

Run on the NSX Manager, or against an extracted NSX support bundle. Substitute the HCX Manager IP address and the HCX NSX service account name from Step 2.

grep "api/2.0/global/config" /var/log/proxy/envoy_access_log.txt*

Expected result: each request logged with a 404 response and an identical small response size.

[<date>T19:36:02.468Z] ##.##.###.# ##.##.###.# "GET" "/api/2.0/global/config" "HTTP/1.1" 404 UAEX 0 131 1 - "##.##.###.#" "Apache-HttpClient/5.5.1" "<request-id>" "<nsx-manager-fqdn>" "-"

Compare the overall response codes from the same HCX Manager address:

grep "<hcx-manager-ip>" /var/log/proxy/envoy_access_log.txt* | grep -oE '" [0-9]{3} ' | sort | uniq -c

Expected result: successful responses dominate, and this path accounts for only a handful of 404 responses.

Confirm the audited NSX operations performed by the HCX service account:

grep "svc-hcx-nsx-####" /var/log/nsx-audit.log* | grep -oE 'Operation="[^"]+", Operation status="[^"]+"' | sort | uniq -c | sort -rn

Expected result: the NSX inventory operations that HCX depends on are recorded as successful, for example:

Operation="ListTier0s", Operation status="success"
Operation="ListTier1", Operation status="success"
Operation="ListSegments", Operation status="success"
Operation="ListAllInfraSegments", Operation status="success"
Operation="ListTransportZonesForEnforcementPoint", Operation status="success"
Operation="ListIpAddressBlocks", Operation status="success"
Operation="ReadClusterConfig", Operation status="success"

These operations supply the transport zones, segments, and gateways used by HCX Network Profiles and Service Mesh. Their success confirms that the NSX integration is functioning and that the legacy path is not required.

Distinguishing this issue from an authentication failure

An expired, locked, or incorrect NSX credential produces a different response. NSX returns HTTP 403 with an error message stating that the credentials were incorrect or the account is locked, and NSX shows a red status indicator in the HCX appliance management interface.

In this article's scenario NSX accepts the credential and returns a 404 stating that the resource path does not exist. If a 403 response and an "incorrect or locked" message are present instead, this article does not apply.

If HCX operations are affected

If Service Mesh, Network Extension, or migration operations fail, a different root cause is present. Contact Broadcom Support for further assistance and provide the following:

1. The exact operation that failed, with the date and time including the time zone.
2. HCX support bundle from the source and target sites, with the HCX database dump option selected.
3. NSX support bundle from the registered NSX Manager cluster.
4. The error text shown in the HCX user interface, transcribed as text.