On-Premises DXO2 Outage: "Tenant does not exist" Error and Pods in CrashLoopBackOff.

Products

• DX Operational Observability

Issue/Introduction

Users are unable to log into the on-premises DXO2 environment and are presented with a "Tenant does not exist" error.

Upon inspecting the Kubernetes environment, the following pods are found in a CrashLoopBackOff state:

• dxi-remediation-container

• ng-acc-configserver-deployment

• dxi-postgres

• dxi-grafana-reporting-engine

• dxi-grafana

Resolution

Login functionality cannot be restored until the DXO2 stack is completely restarted.

To determine the root cause of the pod failures and the tenant error, you must gather specific log files and open a support case with Broadcom.

Steps to proceed:

1. Gather Node Logs: Collect the message logs directly from the affected nodes.

   • Path: /var/log/messages

2. Gather Pod Logs: Collect the logs for the failing pods from the NFS share.

   • Path: /<DXO2 NFS>/oi/ (Replace <DXO2 NFS> with your specific NFS mount directory)

3. Contact Support: Reach out to Broadcom Support for assistance and provide the collected log bundles for analysis.