Aria Automation Requests stuck on Initialization

Products

• VMware Aria Suite

Issue/Introduction

Requests in Aria Automation reach and then get stuck at the Initialization stage

provisioning-service-app logs show  message: Service not found: http://10.244.###.###:8282/provisioning/resource-enumeration-tasks

2024-09-05T14:00:27.265Z WARN provisioning [host='provisioning-service-app-######' thread='xn-index-queries-16' user='' org='' trace='' parent='' span=''] c.v.xenon.common.ServiceErrorResponse.create:83 - message: Service not found: http://10.244.###.###:8282/provisioning/resource-enumeration-tasks/################, statusCode: 404, serverError Id: ################

 use th below command to confirm IP returned in the above does not exist or used in any of the pods:, 

kubectl get services --all-namespaces | grep '10.244.###.###'

Environment

Aria Automation 8.x

Cause

It is possible for pods in Aria Automation to hold on to old kubernetes service IPs that are no longer present in the environment, causing request related activities to hang as resource enumeration cannot complete

Resolution

Resolution to this involves making changes to the appliance. it is recommended to take snapshots of the appliance via LCM > vra environment > the 3 dots > take snapshot prior to making any changes in the appliance. Aria Automation will be unavailable as your run through these steps.  

1. SSH to an Aria Automation appliance

2. Verify the AGE of the KUBE-SYSTEM pods is less than (30) days.

   1. Log into via SSH any of the Aria Automation node(s)

   2. Run this command

     kubectl get pods -n kube-system

   3. If pods AGE are greater then 30 days, restart the Kube-system namespace pods with  

   kubectl delete pod -n kube-system --all 

   4. Run the following command: /opt/scripts/deploy.sh