500 internal server errors in ABX actions after upgrade to VCF Automation 9.1.0.0

Products

• VCF Automation

Issue/Introduction

Post-upgrade to VCF Automation 9.1.0.0, ABX actions intermittently fail with 500 Internal Server Errors when routing outbound traffic through the internal proxy. Server provisioning workflows are impacted, with connections failing during execution.

Symptoms include:

• Intermittent ABX action failures.
• Traceback logs indicating: urllib.error.URLError: <urlopen error Tunnel connection failed: 500 Internal Server Error>
• Full error string: requests.exceptions.ProxyError: HTTPSConnectionPool(host='<host_name>', port=443): Max retries exceeded with url: /api/v1/ip/vra_ranges_include_non_vlans  (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 500 Internal Server Error')))
• Squid-proxy logs displaying: ERROR: Connection to proxy-service.vmsp-platform.svc.cluster.local failed and Detected DEAD Parent: proxy-service.vmsp-platform.svc.cluster.local

Environment

VCF Automation 9.1.0.0

Cause

A legacy configuration artifact from an Aria Automation 8.x import exists in the configuration database. Specifically, a stale callbackReference within an ExtensibilitySubscription document contains an invalid or "rogue" IP address. This IP was incorrectly added to the internal CIDR used by the Virtual Management Services Platform (VMSP) during upgrade, causing the proxy-service to misidentify traffic paths and fail intermittent connection attempts.

Resolution

Prerequisites: 

Identify the Control Plane node and VCF Automation (VCFA) Appliance FQDN

1. Log in to the VCF Operations UI as local admin.
2. Navigate to Build > Lifecycle > VCF Management > Components.
3. Click on VCF Automation
4. Under the VCF Automation component Nodes section, identify the active control plane node and note the IP address (used in Resolution Step 1 as <VCFA-ControlPlane-IP>)
5. Under the VCF Automation component Platform Network Parameters section, identify the VCF Automation FQDN (used in Resolution steps 3 and 4  as <VCFA-FQDN>)

---

Resolution:

1. Open a terminal or SSH client and connect to the control plane node using the vmware-system-useraccount
   
   

   ssh vmware-system-user@<VCFA-ControlPlane-IP>

   (Enter the configured administrative password for vmware-system-user when prompted.)

   
   
   

2. Once connected via SSH, elevate your shell session to superuser status:

   sudo -i

    

3. Export the KUBECONFIG to interact with the cluster: 
   
   export KUBECONFIG=/etc/kubernetes/admin.conf


4. Retrieve the authentication token from the provisioning-secret: 
   
   

   TOKEN=$(kubectl get secrets -n prelude provisioning-secret -o json \
     | jq -r '.data | map_values(@base64d) | .accessToken')

   
   
   
5. Inspect (confirm the stale callbackReference resource):
   
   

   curl -k -H "authorization: BEARER $TOKEN" \
     "https://<VCFA-FQDN>/provisioning/config/extensibility-subscriptions/create_external_subscription:provisioning-service.resource.cud:External-Resource-Sync-Subscription" | jq

    

   
   IMPORTANT NOTE:
   
   If the command returns a HTTP Status: 404 error, first ensure you have entered the VCFA FQDN and not any other IP or FQDN.
   
   If the command returns the message: "Service not found"..."statusCode": 404, this means the stale entry does not exist and VCFA is not affected by this issue.

   
   
   
6. Delete the stale resource:
   
   

   curl -k -H "authorization: BEARER $TOKEN" -X DELETE \
     "https://<VCFA-FQDN>/provisioning/config/extensibility-subscriptions/create_external_subscription:provisioning-service.resource.cud:External-Resource-Sync-Subscription" | jq

   

7. Restart the provisioning-service-app and proxy-service deployments in the prelude namespace to force a refresh of the subscription set:
   
   kubectl rollout restart deployment/provisioning-service-app -n prelude

 kubectl rollout restart deployment/proxy-service -n prelude 


8. Test ABX actions for provisioning to ensure the issue is resolved

Additional Information

To receive updates on this or related issues, you can subscribe to notifications by following the steps in this guide: Subscribe to a Broadcom knowledge article by article or product.