Error: "Failed to start task: Unexpected HTTP response - Status: 500 INTERNAL_SERVER_ERROR" when deploying Cloud Proxy through VCF Fleet 9.1

Products

• VCF Operations

Issue/Introduction

When attempting to deploy a new Cloud Proxy using VCF Fleet 9.1, the deployment task fails and returns the following error message:

Failed to start task (error may be truncated): Unexpected HTTP response (Reference ID: <ID>) - Status: 500 INTERNAL_SERVER_ERROR, reason: IPv4 address '##.##.##.##' is not within the subnet defined by provided gateway '##.##.##.##' and netmask '##.##.##.##' (prefix /25), resolution: Check server logs for details.

Environment

VMware Cloud Foundation (VCF) Operations 9.1

Cause

When deploying an OVA component via the VCF Fleet UI, the interface only prompts for the Fully Qualified Domain Name (FQDN). Fleet then automatically locates the primary VCF Operations VMs and inherits the remaining network configurations (such as the gateway and netmask) from them.

This inheritance causes a subnet mismatch error if you are attempting to deploy the new Cloud Proxy on a different network than the primary VCF Operations instance.

Resolution

Currently, the VCF Fleet UI only supports deploying a Cloud Proxy onto the same network as the VCF Operations instance. To work around this limitation and deploy the proxy on a different network, you can use one of the two methods below:

Method 1: Direct vCenter Deployment

You can bypass the Fleet UI entirely by deploying the Cloud Proxy directly onto the vCenter using an OVA.
Reference: Deploy Cloud Proxy as part of a VCF 9.1 Upgrade

Method 2: Fleet API Deployment

You can deploy the Cloud Proxy using the Fleet API, which allows you to explicitly define the distinct network settings. Run the following steps inside the SDDC Manager or VCF Installer. (Note: Log in as the vcf user and elevate to root using su -).

Prerequisites:

 Set the following variables before running the API commands. You can find the required FQDNs in the VCF Operations UI under Build → Lifecycle → Components:

export OPS_FQDN="<vcf-operations-fqdn>" # Replace with VCF Operations FQDN
export OPS_USER="admin" 
export OPS_PASS=$(systemd-ask-password) # Prompts securely for the Ops admin password
export FLEET_FQDN="<fleet-fqdn>" # Replace with value from VCF Operations UI

1. Get the Fleet LCM Bearer Token
   Fleet LCM APIs require a bearer token issued by VCF Operations.
   
   
   1. Acquire the Ops Session Token:
      
      export OPS_TOKEN=$(curl -ks -X POST \
  "https://${OPS_FQDN}/suite-api/api/auth/token/acquire" \
  --header 'Content-Type: application/json' \
  --data "$(jq -n \
    --arg user "$OPS_USER" \
    --arg pass "$OPS_PASS" '{
      "username": $user,
      "authSource": "LOCAL",
      "password": $pass
  }')" | jq -r '.token') 
echo "Ops Token: $OPS_TOKEN"
      
      
   2. Look up the Fleet LCM service key:
      
      export FLEET_LCM_KEY=$(curl -ks --location --request GET \
  "https://${OPS_FQDN}/casa/services" \
  --header "Authorization: Basic $(printf '%s:%s' "$OPS_USER" "$OPS_PASS" | base64)" \
  | jq -r '.. | objects | select(.type? == "VCF_FLEET_LCM") | .key')
echo "Fleet LCM Key: $FLEET_LCM_KEY"
      
      
   3. Exchange for the Fleet LCM Bearer Token:
      
      export TOKEN=$(curl -ks -X POST \
  "https://${OPS_FQDN}/suite-api/api/auth/token/exchange" \
  --header 'Content-Type: application/json' \
  --header "Authorization: OpsToken ${OPS_TOKEN}" \
  --data "$(jq -n --arg key "$FLEET_LCM_KEY" '{"serviceKeys": [$key]}')" | \
  jq -r '.jwtToken')
echo "Fleet Token: $TOKEN"
      
      
2. Get the VCF Operations Component ID:
   
   curl -ks -X GET \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer ${TOKEN}" \
  "https://${FLEET_FQDN}/fleet-lcm/v1/components" | \
  jq '.components[] | select(.componentType == "OPS") | {id, componentType, fqdn}'
   
   Expected response format in JSON as follows:
    
   {
  "id": "######-#####-####-####-##########",
  "componentType": "OPS",
  "fqdn": ",<VCFOPS_FQDN>" 
}
   
   Save the returned ID as a variable:
   
   export COMPONENT_ID="<ops-component-id>" # Replace with the ID retrieved above
   
   
   
3. Deploy the Cloud Proxy:
   

   Use a PATCH request to deploy the proxy.

   Important Note: When deploying on a different network, you must explicitly provide the gateway and netmask within the ipv4Settings. If omitted, Fleet defaults to the VCF Operations network settings and the deployment will fail. Set addressType to "Static" if using a static IP.

   curl -ks -X PATCH \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer ${TOKEN}" \
  "https://${FLEET_FQDN}/fleet-lcm/v1/components/${COMPONENT_ID}" \
  -d '{
    "deploymentType": "OvaComponentUpdateSpec",
    "componentType": "OPS",
    "nodeSpecs": [
      {
        "nodeType": "ONE_WAY_REMOTE_COLLECTOR",
        "deploymentSpec": {
          "fqdn": "<cloud-proxy-fqdn>",
          "password": "",
          "deploymentOption": "",
          "dnsServers": "",
          "ntpServers": "",
          "dnsSuffix": "",
          "networkName": "",
          "ipv4Settings": {
            "addressType": "Static",
            "address": "<cloud-proxy-ip>",
            "gateway": "<gateway-ip>",
            "netmask": "<subnet-mask>"
          }
        }
      }
    ]
  }'
   
   
4. Monitor the Deployment:
   
   The PATCH response from Step 3 will include a task ID. You can poll this ID to track the deployment progress:
   
   export TASK_ID="<task-id-from-patch-response>" 
curl -ks -X GET \
  --header "Authorization: Bearer ${TOKEN}" \
  "https://${FLEET_FQDN}/fleet-lcm/v1/tasks/${TASK_ID}" | \
  jq
   
   
   A successful deployment will return "status": "SUCCEEDED". You can also monitor the progress via the VCF Operations UI under Build → Lifecycle → Tasks.