Error Code: 'LCMVRAVACONFIG90030, VMware Aria Automation VA Upgrade Status Check failed' while applying Aria Automation 8.18.1 patch

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Aria Suite Lifecycle Manager > Request > request shows failed for Aria Automation 8.18.1 patch with below error code.
  
  
• Error Code: LCMVRAVACONFIG90030
VMware Aria Automation VA Upgrade Status Check failed.
Upgrade on vRealize Automation VA <FQDN> failed with state 'fatal'. Un-expected status in upgrade status check response. To know more about the failure, run command 'vracli upgrade status --details' on the VMware Aria Automation VA <FQDN>." To revert to snapshot, click RETRY with 'revertSnapshot' property set to true.
  
  
• /var/log/vmware/prelude/upgrade-*.log  shows below error referencing to ccs-k3s-app pod.

[ERROR][<TimeStamp>] Pod: ccs-k3s-app-<PoD-ID>is not in Ready or Completed state. All pods must be in either of these states.
[ERROR][<TimeStamp>] Services verification found errors.

• Describing the pod using kubectl describe pod -n prelude ccs-k3s-app-<PoD-ID> on Aria Automation appliance shows pod not found.

• This issue could arise due to any pod "not in Ready or Completed state".
  
  

Environment

Aria Automation 8.18.1

Cause

• The upgrade process failed during the services verification step because the ccs-k3s-app pod or any other pod which was not in a Running state as required. 
• Subsequent investigation using kubectl describe pod indicated that the pod was not found, suggesting it may have been deleted, failed to deploy, or never initialized. 
  This absence of the required pod led to the upgrade script reporting errors and halting the upgrade process.

Resolution

1. Verify Pod Status:
   
   Run the following command to check the pod that is not in running state:
   

   kubectl get pods -n prelude

2. Restart the Pod (if not running)
   
   Note: If the pod is in a running state move to step-4:

   kubectl -n prelude delete pod <Pod_name-ID>

3. Confirm Pod is Running
   
   Ensure the pod is recreated and in a Running state:
   

   kubectl get pods -n prelude

4. Resume the Upgrade
   
   Execute the following command on the Aria Automation appliance (Primary Node) to resume the upgrade and simultaneously retry the upgrade request from LCM UI.

   vracli upgrade exec --resume

5. If you still encounter the same error post the pods are in running state, you can run inventory sync of Aria Automation and restart the patching process
   
   
6. Upgrade Completion
   
   The patch installation should now complete successfully and be marked as complete in Aria Suite Lifecycle Manager.

Additional Information

Post following the steps if the patch history under vRSLCM > Environments > [Environment Name] > Day 2 Tasks > Patches > Patch History does not reflect the current patch level. Review Error LCMVRAVACONFIG900309 is seen during Upgrade/Patch due to Timeout on Aria Automation