Aria Automation Pod Instability and Upgrade Failures Caused by Orphaned Custom Profiles

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

 Orphaned custom profiles in the vmware-prelude framework can cause significant issues in Aria Automation environments.
These orphaned configurations act as "drop-in" overlays that can inadvertently enforce incorrect resource limits on system services, leading to runtime pod instability (e.g., OOMKilled/CrashLoopBackOff) or preventing successful platform upgrades.

Symptoms:

1. System services (such as vco-app or vco-server-app) continuously restart.
2. Pods report "OOMKilled" or "Exit Code 137" errors in kubectl describe outputs.
3. Resource limits and requests displayed in kubectl describe pod do not match expected system profile defaults (e.g., XL profile).
4. Upgrade processes fail at the "perform component upgrade" stage due to missing ConfigMaps or namespace resolution failures.

Environment

• Aria Automation 8.x

Cause

 Orphaned configuration directories located in /etc/vmware-prelude/profiles/ (e.g., custom-profile.old) persist after manual modifications or incomplete profile updates. These files are referenced by the vmware-prelude deployer, causing services to initialize with legacy or incompatible resource constraints.

Resolution

To resolve this issue, you must remove the orphaned profiles and trigger a redeployment of the affected services.

1. Log in to the Aria Automation node via SSH and switch to root: ssh vmware-system-user@<node-ip> sudo -i

2. Identify the orphaned profile directory. Navigate to /etc/vmware-prelude/profiles/ and confirm if any unexpected custom profile directories exist (e.g., custom-profile.old).

3. Relocate the orphaned directory to a temporary location to verify remediation: mv /etc/vmware-prelude/profiles/custom-profile.old /var/tmp/

4. Execute the deployment script to revert services to default profile configurations: /opt/scripts/deploy.sh

5. Monitor the pod status to ensure they restart with the correct resource limits: kubectl describe pod vco-app -n prelude

Note: The removal of profiles triggers a redeployment of services, which typically takes 15–30 minutes.