Error: VCO pod OOM killed 137 causing provisioning pod restarts in VMware Aria Automation

Products

• VCF Automation

Issue/Introduction

The provisioning-service-app pods in Aria Automation are periodically restarted by Kubernetes with an OOMKilled exit code 137. One or more of the following symptoms may be present:

• Daily or frequent restarts of the provisioning-service pods
• kubectl describe pod shows the container terminated with reason OOMKilled
• No OutOfMemoryError is found in the provisioning service logs
• Scaling up the environment to the XL profile does not resolve the restarts
• Provisioning or deployment failures occur during pod restart windows

Environment

Aria Automation 8.18.x

Cause

This issue is caused by a memory leak in the Netty library used within the provisioning service.

Resolution

Fixed in VMware Aria Automation release 8.18.1 P4 and higher.

1. Reference Link:
   VMware Aria Automation 8.18.1 Update 4

2. Follow the documented procedure to Scale up VMware Aria Suite products if resources are still constrained.
   
   • Note: Manually resizing CPU or memory through vCenter is not supported and will not correctly resize the pod deployments.

Workaround:

• Configure Netty to use heap memory by utilizing custom profiles (survive re-deploys and upgrades preserving the change)
• Take simultaneous VM snapshots (without memory) of all vRA nodes as a precaution.
• If a previous version of the custom profile was already applied, remove it first by running the following on one vRA node:

  vracli cluster exec -- bash -c 'rm -rf /etc/vmware-prelude/profiles/netty-no-direct-memory'
  

• Create the custom profile by executing the following command on one vRA node:

  vracli cluster exec -- bash -c 'current_node; base64 -d <<< IyEvYmluL2Jhc2gKIyBDb3B5cmlnaHQgKGMpIDIwMjYgQnJvYWRjb20uIEFsbCBSaWdodHMgUmVzZXJ2ZWQuCiMgQnJvYWRjb20gQ29uZmlkZW50aWFsLiBUaGUgdGVybSAiQnJvYWRjb20iIHJlZmVycyB0byBCcm9hZGNvbSBJbmMuCiMgYW5kL29yIGl0cyBzdWJzaWRpYXJpZXMuCgojIENyZWF0ZSBjdXN0b20gcHJvZmlsZSBkaXJlY3RvcnkKbWtkaXIgLXAgL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9uZXR0eS1uby1kaXJlY3QtbWVtb3J5LwoKIyBDcmVhdGUgdGhlIHJlcXVpcmVkIGRpcmVjdG9yeSB0cmVlIHRoYXQgd2lsbCBiZSB1c2VkIHdoZW4gdGhlIHByb2ZpbGUgaXMgYWN0aXZlCm1rZGlyIC1wIC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMvbmV0dHktbm8tZGlyZWN0LW1lbW9yeS9oZWxtL3ByZWx1ZGVfcHJvdmlzaW9uaW5nLXNlcnZpY2UvCgojIENyZWF0ZSAiY2hlY2siIGZpbGUgdGhhdCBpcyBhbiBleGVjdXRhYmxlIGZpbGUgcnVuIGJ5IGRlcGxveSBzY3JpcHQKY2F0IDw8RU9GID4gL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9uZXR0eS1uby1kaXJlY3QtbWVtb3J5L2NoZWNrCiMhL2Jpbi9iYXNoCmV4aXQgMApFT0YKY2htb2QgNzU1IC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMvbmV0dHktbm8tZGlyZWN0LW1lbW9yeS9jaGVjawoKIyBDb3B5IHByb3Zpc2lvbmluZy1zZXJ2aWNlIGNvbmZpZ3VyYXRpb24gZmlsZSB0byB0aGUgY3VzdG9tIHByb2ZpbGUKY2F0IDw8RU9GID4gL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9uZXR0eS1uby1kaXJlY3QtbWVtb3J5L2hlbG0vcHJlbHVkZV9wcm92aXNpb25pbmctc2VydmljZS85MC1yZXNvdXJjZXMueWFtbApuZXR0eU5vUHJlZmVyRGlyZWN0OiB0cnVlCm1heERpcmVjdE1lbW9yeVNpemU6ICIyNTZtIgpqYXZhT3B0czoKICAiaW8ubmV0dHkubWF4RGlyZWN0TWVtb3J5IjogIjAiCkVPRgpjaG1vZCA2NDQgL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9uZXR0eS1uby1kaXJlY3QtbWVtb3J5L2hlbG0vcHJlbHVkZV9wcm92aXNpb25pbmctc2VydmljZS85MC1yZXNvdXJjZXMueWFtbAo= | bash -'

   

• Apply the profile by running the following on one vRA node:

  /opt/scripts/deploy.sh

• The provisioning-service pods will restart automatically to apply the new settings. No further action is required across other nodes - the profile propagates via vracli cluster exec.

Verify after the restart, monitor pod stability:

• Check pod status and restart counts

  kubectl -n prelude get pods -l app=provisioning-service-app

• Inspect pod events for OOMKilled signals

  kubectl -n prelude describe pod <provisioning-service-app-pod-id>

Revert (if needed):

• To remove the custom configuration:
• Run on one vRA node

  vracli cluster exec -- bash -c 'rm -rf /etc/vmware-prelude/profiles/netty-no-direct-memory'

• Run the deploy command

  /opt/scripts/deploy.sh

 

 

Additional Information

Note:

If vco-app (vRO) pods are also restarting frequently after applying Aria Automation 8.18.1 Patch 4, This is can be addressed by upgrading the vSphere Plugin to version 8.2.1

Reference KB: 432637