How to recover VMware Integrated OpenStack if the admin project is deleted

Products

• VMware
• VMware Integrated OpenStack

Issue/Introduction

Symptoms:
If the admin project of the OpenStack deployment is deleted we will run into errors with some OpenStack commands ran from the toolbox of VMware Integrated OpenStack 6.x and 7.x. The error message will be similar to the following:
The request you have made requires authentication. (HTTP 401) (Request-ID: req-<UUID>)

Environment

VMware Integrated OpenStack 6.x
VMware Integrated Openstack 7.x

Cause

The admin project would have to have been manually deleted by an end-user or administrator. The project is critical to the operations of VMware Integrated OpenStack.

Resolution

1. SSH to the VIO manager as root.
2. Delete the keystone helm chart with: helm delete keystone1 --purge
3. Find the keystone helm jobs: osctl get job|grep helm-ke
4. Delete the helm keystone jobs: osctl delete job <helm-keystone-keystone1-UUID>
5. Wait for the pods to rebuild, and re-attempt the OpenStack Client commands. The project should be rebuilt.

Additional Information

Impact/Risks:
The resolution steps do create momentary downtime that should be resolved from 1-10 minutes depending on the environment.