AI Assistant Platform status is degraded or down

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention
• VMware Avi Load Balancer

Issue/Introduction

You are running SSP or Avi Operations and have encountered an alarm with any of the following description:

AI Assistant Platform {{ .ResourceID }} is degraded
AI Assistant Platform {{ .ResourceID }} is down

The {{ .ResourceID }} can be any of the following:

1. 1. intelligent-assist-browser

   2. intelligent-assist-edge

   3. intelligent-assist-models

   4. intelligent-assist-nginx

   5. intelligent-assist-search

   6. intelligent-assist-tutor

   7. intelligent-assist-vdefend-dialog-0

Environment

vDefend SSP Version: 5.2 and later

Avi Operations: 5.2 and later

Cause

 

• Resource Exhaustion: High CPU, memory, or disk usage that causes the node to become unresponsive or unstable.

• Kubernetes Node Failures: Issues with Kubernetes node components such as the kubelet, or node being marked as "NotReady" due to insufficient resources, failed pods, or crashed components.

 

Resolution

Steps to resolve:

1. Check AI Assistant disk usage, memory usage, CPU usage alarms.

2. Try re-starting the deployment/statefulset. This should take care of any transient issues. This will impact any ongoing chat conversations.

3. Log into SSP-Installer using 'sysadmin' credentials

4. Get the pod name for the {{ .ResourceID }} , using 'k -n nsxi-platform get pods | grep {{ .ResourceID }}'

k -n nsxi-platform get pod <pod-name> -o jsonpath='{.metadata.ownerReferences[0].kind}'

5. Confirm the output is ReplicaSet and run:

k -n nsxi-platform rollout restart deployment {{ .ResourceID }} 

6. If none of the above is applicable, please open a ticket with Broadcom Support.