Heat stack deletion fails with "security group ##-##-##-##-## in use" neutron conflict error

Products

• VMware Integrated OpenStack

Issue/Introduction

Heat stack deletions fail and permanently revert to a DELETE_FAILED status. Standard automated deletion loops using openstack stack delete fail to complete cleanly. The heat-engine logs show a Conflict exception thrown by the Neutron networking subsystem when attempting to strip out active security group configurations:

heat-engine.log

### ## ##:##:## controller_########### heat-engine[###]: ####-##-## ##:##:##:### 1 DEBUG neutronclient.v2_0.client [req-########-####-####-####-###########- Performance - ###########################] Error message: {"NeutronError": {"type": "SecurityGroupInUse", "message": "Security Group <REDACTED_SECRET> in use.", "detail": ""}}
### ## ##:##:## controller_########### heat-engine[###]: ####-##-## ##:##:##:### 1 ERROR heat.engine.resource neutronclient.common.exceptions.Conflict: Security Group <REDACTED_SECRET> in use.
### ## ##:##:## controller_########### heat-engine[###]: ####-##-## ##:##:##:### 1 INFO heat.engine.stack [req-########-####-####-####-############] Stack DELETE FAILED: Resource DELETE failed: Conflict: resources.zook-client-sg: Security Group <REDACTED_SECRET> in use.

### ## ##:##:## controller-########## heat-engine[###]: ####-##-## ##:##:##:### 1 DEBUG heat.db.sqlalchemy.api [req-########-####-####-####-############- Performance - ########################### #######################] Did not set stack state with values {'action': 'DELETE', 'status': 'FAILED', 'status_reason': "Resource DELETE failed: Conflict: resources.zook-client-sg: Security Group ########-####-####-############### in use.\nNeutron server returns request_ids: ['req-######-####-####-####-############']"}, stack id: ########-####-####-####-############ with expected traversal: ########-####-####-####-########## stack_update /usr/lib/python3.7/site-packages/heat/db/sqlalchemy/api.py:###
### ## ##:##:## controller-k5h9tvfrch heat-engine[###]: ####-##-## ##:##:##:### 1 INFO heat.engine.stack [req-########-####-####-####-############# - Performance - ####################### ###################] Stack sit-test1-alerts-security-groups-common-rules-s25du5jfmaas traversal ########-####-####-####-############ no longer active; not setting state to DELETE_FAILED

Environment

VIO: 7.3

Cause

The primary cause is an active database-level relational lock within the OpenStack Neutron networking subsystem, where continuing port bindings or remote group rules maintain a reference constraint on the unique identifier of the security group.

Resolution

Steps followed to remove the stack:

1. Extract the Root Password from the Container Environment.
   
   
   1. Execute the following command on the VIO Manager to print out the active environment strings and isolate the database administrative password variable:
      
      

      # osctl exec -it mariadb-server-0 -n openstack -- env | grep -i PASSWORD

   2. Identify the string assigned to DB_ROOT_PASSWORD or MARIADB_ROOT_PASSWORD.
      
      
2. Identify Conflicting Port Mappings.
   
   
   1. Run this command to check for any persistent logical ports bound directly to the targeted Security Group UUID:
      
      

      # osctl exec -it mariadb-server-0 -n openstack -- bash -c 'mysql -u root -p$DB_ROOT_PASSWORD -e "SELECT port_id FROM neutron.securitygroupportbindings WHERE security_group_id=\"########-####-####-####-##############\";"'

       

   2. Output from the command "openstack stack event list <stack name>:

      ### ## ##:##:##controller-########### heat-engine[###]: ####-##-####:##:##.### 1 DEBUG heat.db.sqlalchemy.api [req-#######-####-####-####-#############- Performance - ################### ##############] Did not set stack state with values {'action': 'DELETE', 'status': 'FAILED', 'status_reason': "Resource DELETE failed: Conflict: resources.zook-client-sg: Security Group #######-####-####-####-#############c in use.\nNeutron server returns request_ids: ['req-#######-####-####-####-#############']"}, stack id: #######-####-####-####-############# with expected traversal: #######-####-####-####-############# stack_update /usr/lib/python3.7/site-packages/heat/db/sqlalchemy/api.py:791

       

3. Identify Interdependency Rules (Remote Group References)
   
   
   1. If the port mapping scan returns an empty set, the block is caused by a rule reference constraint where another security group utilizes this group as its remote traffic source. Run this command to isolate the blocking rule entry:
      
      

      # osctl exec -it mariadb-server-0 -n openstack -- bash -c 'mysql -u root -p$DB_ROOT_PASSWORD -e "SELECT id, security_group_id FROM neutron.securitygrouprules WHERE remote_group_id=\"########-####-####-####-##############\";"' 

      
      Note: Iterate through the project IDs or names discovered in the previous step to locate where the security group is bound. Prioritize administrative and infrastructure service projects (such as admin or service):

      # openstack port list --project <PROJECT_ID_OR_NAME> --long | grep ########-####-####-####-##############

   2. Run the following command using the discovered Port UUID to extract its network attachment properties, host bindings, and device ownership:

      openstack port show <PORT_ID> -c device_id -c device_owner -c fixed_ips -c network_id

      
      
      
4. If a Port ID is Returned.
   
   
   1. Delete the network port holding the resource lock via the OpenStack CLI:
      
      

      openstack port delete <PORT_ID>

       
5. Re-run the Heat stack:
   
   

   openstack stack delete ###-######-#####-##-######-######-####### --wait --yes

Additional Information

Unable to Delete Stacks Stuck in "DELETE_FAILED" State

ERROR: Resource DELETE failed: MessagingTimeout: resources.evan-cluster: Timed out waiting for a reply to message ID <message UUID>