HCX 9.x SecretRotationJob generates HTTP 400 Bad Request against NSX 4.x

Products

• VMware HCX

Issue/Introduction

In environments using VMware Cloud Foundation (VCF) Operations 9.x, VMware HCX 9.x, and legacy VMware NSX 4.x, a continuous error loop is observed.

• The HCX SecretRotationJob fails to retrieve policies from the NSX Manager.
• The following errors are observed in /common/logs/admin/app.log:

  <timestamps> ERROR hcx-app-engine 3110 [hcx@4413 logger="c.v.h.s.s.AbstractSecreteRotationJob" thread="SecretRotateService_SvcThread-1" ent="Ent: HybridityAdmin" user="" txid="#########-####-####-####-############"] Error running the SecretRotationJob, failing this (jobId:#########-####-####-####-############) and initiating new job with delay. Error:
  java.lang.RuntimeException: Failed to get NSX-T policy for service account '<username>'. API Response: {"status":"failure","statusCode":400,"details":"","result":{"error_code":255,"error_message":"One or more query strings not valid: 'filter=all' - invalid property 'filter': properties not specified by schema are not allowed by NoRestRequestParameters","module_name":"common-services"}}

Environment

VMware Cloud Foundation (VCF) Operations 9.x
VMware HCX 9.x
VMware NSX 4.x

Cause

VCF Operations HCX 9.x introduces an NSX service account password rotation feature.
The SecretRotateService starts the NsxtSecretRotationJob automatically upon detecting an SDDC Manager 9.x configuration.

During the policy check state, the job issues an API request (GET https://<HOSTNAME>/api/v1/node/users?filter=all) to the NSX Manager.
The ?filter=all query parameter is required in NSX 9.x to include service-type accounts, but it does not exist in the NSX 4.x API schema. NSX 4.x rejects the request with an HTTP 400 Bad Request. 

Resolution

This is a known issue impacting VCF Operations HCX 9.x.

Upgrade the VMware NSX environment to version 9.x to natively support the required API schema and query parameters.