VIO Octavia loadbalancer stuck in a PENDING_UPDATE status

Products

• VMware Integrated OpenStack

Issue/Introduction

• Load Balancer can not be deleted
• Load Balancer is in a PENDING_UPDATE status.
  
  Error message:
  OctaviaClientException: Invalid state PENDING_UPDATE of loadbalancer resource ######-####-####-####-######### (HTTP 409) (Request-ID: req-########-####-####-####-#########)

Environment

7.x

Cause

Currently, there is no built-in mechanism to change the load balancer into an error state for easy removal through the User Interface.

Resolution

Note: This procedure modifies the database. Ensure you to take a backup of the database before proceeding.  Do not skip this step.

Backup an Openstack Database

1. osctl exec -it mariadb-server-0 bash
2. mysqldump --defaults-file=/etc/mysql/admin_user.cnf --host=localhost  octavia > /tmp/octavia.sql
3. exit 
4. osctl cp openstack/mariadb-server-0:/tmp/octavia.sql /tmp/octavia.sql
5. Then pull the backup from the manager.  Using something like winscp

Workaround:

1. Make sure no object exists for these LBs in NSX (search by their Openstack id using NSX UI). If any object present in NSX, clean them up as well.
2.  Log in to the LCM node and log in to a mariadb pod:
   
   kubectl -n openstack exec -it mariadb-server-0 bash

3. Find the loadbalancer id in the ocatavia db and update its status.
   
   Example:
   
   # mysql --defaults-file=/etc/mysql/admin_user.cnf --host=localhost
# use octavia;
# select * from load_balancer;
# update load_balancer set provisioning_status='ERROR' where id = "########-####-####-####-#####";

4. Delete the loadbalancer in error state using openstack CLI:

openstack loadbalancer list --operating-status=ERROR
openstack loadbalancer delete <load balancer ID>