"Error: Local-service has stateful services and cannot be deleted. Retry after removing services: [Service Insertion]. (Error code: 503109) " when deleting the Edge Gateway

Products

• VMware NSX

Issue/Introduction

• You may encounter an error when attempting to delete a Tier-1 Gateway in NSX. The deletion fails with the following error message displayed in the NSX Manager UI:

Error: Local-service Locale Service '<Locale-Service_Name>' has stateful services and cannot be deleted. Retry after removing services: [Service Insertion]. (Error code: 503109)

• You may have applied resolution provided in the KB Error: "Bad Request: Error occurred in the backing network provider...error code 503109" when deleting the Edge Gateway from VCD
• In NSX Manager /var/log/proton/nsxapi.log, the following entries are present:

ERROR http-nio-127.0.0.1-7440-exec-1223 LocaleSvcServiceImpl 77338 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM503109" level="ERROR" reqId="27dd53bd-9ad6-43a3-b1f0-c37d1676429f" subcomp="manager" username="admin"] Local-service /infra/tier-1s/<tier-1_ID>/locale-services/<locale-service_ID> has stateful services and cannot be deleted. Retry after removing services: [Service Insertion].
WARN http-nio-127.0.0.1-7440-exec-1223 TransactionRetryAspect 77338 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="27dd53bd-9ad6-43a3-b1f0-c37d1676429f" subcomp="manager" username="admin"] void com.vmware.nsx.management.policy.connectivity.service.LocaleSvcServiceImpl.delete(PolicyPath) failed with class com.vmware.nsx.management.policy.connectivity.common.exceptions.ConnectivityIntentException.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The deletion of the T1 Gateway is blocked because the associated Locale Service still has a stateful service configured.

Resolution

Please perform the following workaround steps if you have encountered this issue

Step 1: Verify Locale Service Reference

Perform an Elasticsearch for the Locale Service ID to confirm that it is not being referenced by any other NSX objects.

 

Step 2: Delete the Locale Service

If the Locale Service is only associated with the T1 Gateway being deleted, you can safely remove it using the following API call:

DELETE https://<Manager_FQDN>/policy/api/v1/infra/tier-1s/<T1_ID>/locale-services/<LocaleService_ID>

Once the Locale Service is deleted successfully, retry deleting the T1 Gateway from the NSX Manager UI or API.

 

Step 3: If Deletion of Locale Service Fails

If the above API call fails, perform a global search in the NSX Manager UI for the Locale Service ID to locate any associated VirtualServiceInstance objects.

Identify the endpoint path similar to:

/infra/tier-1s/<tier-1_ID>/locale-services/<locale-service_ID>/endpoints/virtual-endpoints/<virtual-endpoint_ID>

Then, remove the Virtual Service association using the following API call:

DELETE https://<Manager_FQDN>/policy/api/v1/infra/tier-1s/<tier-1_ID>/locale-services/<locale-service_ID>/endpoints/virtual-endpoints/<virtual-endpoint_ID>

Once you receive a 200 OK response, reattempt the T1 Gateway deletion.