Error: "Bad Request: Error occurred in the backing network provider...error code 503109" when deleting the Edge Gateway

Products

• VMware Cloud Director

Issue/Introduction

• Unable to delete the Edge Gateway due the error below:
  
  [ ########-####-####-########9102 ] Bad Request: Error occurred in the backing network provider: Local-service path=[/infra/tier-1s/########-####-####-########b42c/locale-services/########-####-####-########ef6e] has stateful services and cannot be deleted. Retry after removing services: [Service Insertion]., error code 503109


• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log the below entries are present:
  
  

  DATE TIME | DEBUG    | task-service-activity-pool-144 | RestClient                     | Retry error message not found "error_code" : 500045 in response for DELETE /policy/api/v1/infra/tier-1s/########-####-####-########b42c/locale-services/########-####-####-########ef6e with status 400 after execution 1:
{
  "error_code" : 503109,
  "module_name" : "Policy",
  "error_message" : "Local-service path=[/infra/tier-1s/########-####-####-########b42c/locale-services/########-####-####-########ef6e] has stateful services and cannot be deleted. Retry after removing services: [Service Insertion]."
} | requestId=########-####-####-########f041,request=DELETE https://cloud.example.com/api/admin/edgeGateway/########-####-####-########b42c,requestTime=1748261698012,remoteAddress=##.##.##.##:52285,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/201...,accept=application/*+json;version 40.0.0-alpha;multisite ########-####-####-########3ca6@########-####-####-########77d2 vcd=########-####-####-########f474,task=########-####-####-########6d7c activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-########6d7c)

  AND 
  
  DATE TIME | DEBUG    | task-service-activity-pool-144 | NsxTEdge                       | Exception occurred while undeploying and deleting edge gateway GATEWAY_NAME (DEL) | requestId=########-####-####-########f041,request=DELETE https://cloud.example.com/api/admin/edgeGateway/########-####-####-########b42c,requestTime=1748261698012,remoteAddress=##.##.##.##:52285,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/201...,accept=application/*+json;version 40.0.0-alpha;multisite ########-####-####-########3ca6@########-####-####-########77d2 vcd=########-####-####-########f474,task=########-####-####-########6d7c activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-########6d7c)
com.vmware.vcloud.common.network.VsmException: Bad Request: Error occurred in the backing network provider: Local-service path=[/infra/tier-1s/########-####-####-########b42c/locale-services/########-####-####-########ef6e] has stateful services and cannot be deleted. Retry after removing services: [Service Insertion]., error code 503109
        at com.vmware.vcloud.fabric.nsm.error.NetworkSecurityErrorHandler.processException(NetworkSecurityErrorHandler.java:103)
        at com.vmware.vcloud.fabric.nsm.error.NetworkSecurityErrorHandler.handleError(NetworkSecurityErrorHandler.java:78)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
[...]

• The Edge Gateway to be deleted still exists in NSX and there are no services and segments linked to it.

Environment

VMware Cloud Director 10.6

Cause

The issue occurs when the services linked to the Edge Gateway have been deleted directly form NSX.

Resolution

Database modification is needed in order to delete the Edge Gateway.

Contact Broadcom Support and note this Article ID (403440) in the problem description. For more information, see Creating and managing Broadcom support cases.