Database deployment fails in Data Services Manager due to AKO(Avi Kubernetes Operator) resource conflict in workload cluster

Products

• VMware Data Services Manager for VCF Private AI Services

Issue/Introduction

Provisioning database clusters in VMware Data Services Manager (DSM) fails with a "Degraded" or "Failed" status.

The database pod remains in a pending state, and provisioning reports StorageReady=false.

This issue occurs when an additional AKO (Avi Kubernetes Operator) resource is deployed in the workload cluster, interfering with the reconciliation process.

Environment

VMware Data Services Manager (DSM)
vSphere Kubernetes Service (vKS)
AVI Load Balancer

Cause

The deployment of an additional AKO resource in the workload cluster disrupts the creation and deletion operations for the database clusters.

The ako.kubernetes.vmware.com/install label set to true causes conflicts during the reconciliation of the database clusters, leading to provisioning errors.

The deployment failed after describe the failed Pod with 'not enough memory to start the pod'

Resolution

To resolve this issue, modify the AKO installation label to prevent the conflict:

1. Establish an SSH session to the Supervisor Cluster.
2. Identify the affected clusters by running the following command:   kubectl get clusters -A
3. Edit the cluster configuration:   kubectl edit clusters -n <namespace_name>
4. Locate the following configuration entry:   ako.kubernetes.vmware.com/install: "true"
5. Change the value to false: ako.kubernetes.vmware.com/install: "false"
6. Save and exit the editor
7. Retrigger the reconciliation of the database clusters by annotating them with a dummy value.

Additional Information

 If the issue persists after following the steps above, please gather logs and contact support.