NSX Advanced Load Balancer Deployment from SDDC Manager Fails to Deploy

Products

VMware SDDC Manager

Issue/Introduction

During the deployment of NSX Advanced Load Balancer (ALB) from the SDDC Manager UI, the operation fails with the following error message:

Message: Failed to create Service users for #### NSX Advanced Load Balancer
Cause: Failed to create Service users for #### NSX Advanced Load Balancer
InvalidRequest (com.vmware.vapi.std.error.invalid_request) => { 
 messages = [], 
 data = struct => {
   error_message = This API is only allowed in VCF/SDDC Managed NSX.,
   httpStatus = BAD_REQUEST,
   error_code = 94514,
   module_name = POLICY
 },
 errorType = INVALID_REQUEST
}

Environment

VMware Cloud Foundation 5.x

Cause

This issue occurs when the SDDC environment has been upgraded across multiple major versions (for example, from 3.x → 4.x → 5.x).

Starting with VMware Cloud Foundation 5.x, a watermarking mechanism was introduced for NSX environments to indicate that NSX is being managed by SDDC Manager.

If the NSX environment was deployed prior to the upgrade to 5.x, it will not have the watermark, causing SDDC Manager to reject API requests for ALB deployment with an InvalidRequest error.

Resolution

Important:
Before making any changes, take a snapshot of the SDDC Manager virtual machine to allow for rollback if required.

Follow the steps below to manually watermark the NSX environment:

Step 1: Verify if NSX is Watermarked

Log in to the SDDC Manager appliance via SSH as the vcf or root user.

Run the following command (replace admin:##### with your NSX Manager credentials and <NSX-IP> with the NSX Manager IP address):

curl -k -s -X GET -u 'admin:#####' https://<NSX-IP>/api/v1/watermark

If the NSX environment is not watermarked, the response will resemble the following output:

{
  "httpStatus": "NOT_FOUND",
  "error_code": 600,
  "module_name": "common-services",
  "error_message": "The requested object : Deployment Info Details could not be found. Object identifiers are case sensitive."
}

Step 2: Manually apply the watermark

To manually watermark the NSX Manager, run the following command (replace placeholder values accordingly):

curl -k -s -X POST -u 'admin:#####' https://<NSX-IP>/api/v1/watermark \
  -H "Content-Type: application/json" \
  -d '{
    "deployed_by": "##########",
    "version": "##########",
    "managed_by": "##########",
    "instance_id": "##########",
    "log_token": "##########"
  }'

The required values can be retrieved from the SDDC Manager PostgreSQL database using the following command:

psql -h localhost -U postgres -d platform -c "select * from sddc_manager_controller;"

version → corresponds to sddc_manager_version
managed_by → corresponds to sddc_manager_fqdn (listed as vm_hostname in the table)
instance_id → corresponds to sddc_manager_id

Step 3: Retry the NSX ALB deployment

Once the watermark has been applied successfully:

Restart the failed ALB deployment task, or

Cancel the existing task and initiate a new NSX Advanced Load Balancer deployment from the SDDC Manager UI.

Additional Information

This issue typically affects environments upgraded from VCF 3.x or 4.x to VCF 5.x where the NSX Manager instance predates the introduction of watermarking.
Always ensure that the SDDC Manager and NSX components are at compatible versions as per the VMware Cloud Foundation Bill of Materials (BOM).

Impact / Risk

If watermarking is not present, SDDC Manager cannot validate ownership of the NSX Manager instance, resulting in deployment or integration failures for components such as NSX ALB.

Applying the watermark manually re-establishes the trust relationship between SDDC Manager and NSX.