Aria Automation 8.18.1 Patch Upgrade Hangs during Docker Image Loading Phase

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

An Aria Automation patch upgrade takes significantly longer than expected or appears to hang without progress during the image-loading phase.

The upgrade log shows no activity for an extended period after the RPM installation completes. This can be observed in the /var/log/vmware/prelude/upgrade.log file between the following messages: Executing post-update script and Post-update script completed successfully.

Docker image loading takes multiple hours. This can be observed in the /var/log/bootstrap/postupdate-patch.log file between the following messages: Loading updated images and Done loading updated images.

The journalctl log from the upgrade period contains the following message: Kubelet is still running. Cleanup cannot be performed.

Aria LCM fails with Error Code: LCMVRAVACONFIG90030 due to a timeout

Environment

Aria Automation 8.18.1

Cause

During the RPM upgrade phase, the Docker service is restarted while the kubelet is still active, which prevents the built-in Docker storage cleanup mechanism from executing. This allows hundreds of thousands of stale Docker network sandbox entries to accumulate in /data/docker/network/files/local-kv.db. Docker attempts to process these stale entries on startup, stalling the post-update image-loading phase and the entire cluster upgrade.

Resolution

A permanent fix will be included in Aria Automation 8.18.1 Patch 6 and later releases. To work around the issue on Patch 4 or Patch 5, deploy a pre-update script to all nodes before initiating the upgrade:

Note: In case if the upgrade is hung for a long time, do revert to pre-upgrade snapshot and implement the below steps before reattempting the patch update.

Take simultaneous VM snapshots (without memory) of all Aria automation nodes.(In case the upgrade is hung do revert to the snapshot and apply these steps)
Run the following command from any cluster node as root before initiating the upgrade to deploy the preupdate.sh script to all nodes simultaneously:

vracli cluster exec -- bash -c "echo bWtkaXIgLXAgL3Zhci92bXdhcmUvcHJlbHVkZS91cGdyYWRlL3J1bgpjYXQgPiAvdmFyL3Ztd2FyZS9wcmVsdWRlL3VwZ3JhZGUvcnVuL3ByZXVwZGF0ZS5zaCA8PCAnUFJFVVBEQVRFX0VPRicKIyEvYmluL2Jhc2gKIwojIFdvcmthcm91bmQgZm9yIHNsb3cgRG9ja2VyIHN0YXJ0dXAgZHVyaW5nIHVwZ3JhZGUKIwojIFJvb3QgY2F1c2U6CiMgICBEdXJpbmcgcGF0Y2ggdXBncmFkZXMsIERvY2tlciBpcyByZXN0YXJ0ZWQgYXMgcGFydCBvZiB0aGUgUlBNIHVwZ3JhZGUgcGhhc2UuCiMgICBjbGVhbnVwX2RvY2tlcl9zdG9yYWdlLnNoIChFeGVjU3RhcnRQcmUgZm9yIGRvY2tlci5zZXJ2aWNlKSBza2lwcyBpdHMgd2lwZQojICAgd2hlbiBrdWJlbGV0IGlzIGFjdGl2ZSwgbGVhdmluZyBzdGFsZSBzYW5kYm94IGVudHJpZXMgaW4gbG9jYWwta3YuZGIgaW50YWN0LgojICAgT24gaGVhdmlseS1sb2FkZWQgZW52aXJvbm1lbnRzIHdpdGggbG9uZyB1cHRpbWVzIChtb250aHMgd2l0aG91dCBhIERvY2tlcgojICAgcmVzdGFydCBvciBub2RlIHJlYm9vdCksIHRob3VzYW5kcyBvZiBzdGFsZSBzYW5kYm94IHJlY29yZHMgYWNjdW11bGF0ZTsKIyAgIERvY2tlciB0aGVuIHNwZW5kcyAxLTExIGhvdXJzIHByb2Nlc3NpbmcgdGhlbSBvbiBzdGFydHVwLCBibG9ja2luZyB0aGUKIyAgIHBvc3R1cGRhdGUgaW1hZ2UtbG9hZGluZyBwaGFzZSBhbmQgc3RhbGxpbmcgdGhlIGVudGlyZSBjbHVzdGVyIHVwZ3JhZGUuCiMKIyBGaXg6CiMgICBTdG9wIGt1YmVsZXQgYmVmb3JlIHRoZSBSUE0gaW5zdGFsbCBzbyBjbGVhbnVwX2RvY2tlcl9zdG9yYWdlLnNoIHNlZXMKIyAgIGt1YmVsZXQgYXMgaW5hY3RpdmUgYW5kIHBlcmZvcm1zIHRoZSBmdWxsIHdpcGUgb2YgL2RhdGEvZG9ja2VyLyogd2hlbgojICAgRG9ja2VyIGlzIHJlc3RhcnRlZCBkdXJpbmcgdGhlIFJQTSB1cGdyYWRlIHBoYXNlLgojICAgVGhlIHN5c3RlbSByZWJvb3QgdGhhdCBmb2xsb3dzIHRoZSB1cGdyYWRlIHJlc3RhcnRzIGt1YmVsZXQgYXV0b21hdGljYWxseS4KIwoKc2V0IC1ldW8gcGlwZWZhaWwKCmxvZ0Rpcj0iL3Zhci9sb2cvYm9vdHN0cmFwIgp0aW1lc3RhbXA9JChkYXRlICcrJVktJW0tJWQtJUgtJU0tJVMnKQpQQVRDSF9QUkVVUERBVEVfTE9HPSIkbG9nRGlyL3ByZXVwZGF0ZS1wYXRjaC0kdGltZXN0YW1wLmxvZyIKCm1rZGlyIC1wICIkbG9nRGlyIgoKZXhlYyA+ID4odGVlIC1hICIkUEFUQ0hfUFJFVVBEQVRFX0xPRyIpIDI+JjEKCmZ1bmN0aW9uIGxvZygpIHsKICBsb2NhbCBtc2c9IiQxIgogIGxvY2FsIGxldmVsPSIkMiIKICBsb2NhbCBkdD0kKGRhdGUgJyslWS0lbS0lZCAlSDolTTolUycpCiAgbG9jYWwgaG9zdE5hbWU9IiQoaG9zdG5hbWUgLWYpIgogIGVjaG8gIlskbGV2ZWxdWyRkdF1bJGhvc3ROYW1lXSAkbXNnIiA+JjIKfQoKZnVuY3Rpb24gbG9nX2luZm8oKSB7CiAgbG9nICIkMSIgIklORk8iCn0KCmZ1bmN0aW9uIGxvZ193YXJuKCkgewogIGxvZyAiJDEiICJXQVJOSU5HIgp9CgpmdW5jdGlvbiBsb2dfZXJyb3IoKSB7CiAgbG9nICIkMSIgIkVSUk9SIgp9Cgpsb2dfaW5mbyAiU3RvcHBpbmcga3ViZWxldCB0byBhbGxvdyBEb2NrZXIgc3RvcmFnZSBjbGVhbnVwIGR1cmluZyBSUE0tdHJpZ2dlcmVkIHJlc3RhcnQgLi4uIgpzeXN0ZW1jdGwgc3RvcCBrdWJlbGV0LnNlcnZpY2UKCmxvZ19pbmZvICJSZW1vdmluZyBydW5uaW5nIGNvbnRhaW5lcnMgdG8gcmVsZWFzZSBvdmVybGF5MiBtb3VudCBsb2NrcyAuLi4iCmlmIHN5c3RlbWN0bCBpcy1hY3RpdmUgLS1xdWlldCBkb2NrZXIuc2VydmljZTsgdGhlbgogICAgZG9ja2VyIHBzIC1hcSB8IHhhcmdzIC1ybjEgZG9ja2VyIHJtIC1mIDI+L2Rldi9udWxsIHx8IHRydWUKZmkKCmxvZ19pbmZvICJLaWxsaW5nIG9ycGhhbmVkIGNvbnRhaW5lcmQtc2hpbSBwcm9jZXNzZXMgLi4uIgpwZ3JlcCAtZiBjb250YWluZXJkLXNoaW0gfCB4YXJncyAtciBraWxsIC05IDI+L2Rldi9udWxsIHx8IHRydWUKCmxvZ19pbmZvICJQcmUtdXBkYXRlIGNvbXBsZXRlLiBrdWJlbGV0IHN0b3BwZWQ7IERvY2tlciBzdG9yYWdlIHdpbGwgYmUgY2xlYW5lZCBvbiBuZXh0IFJQTS10cmlnZ2VyZWQgcmVzdGFydC4iClBSRVVQREFURV9FT0YKY2htb2QgK3ggL3Zhci92bXdhcmUvcHJlbHVkZS91cGdyYWRlL3J1bi9wcmV1cGRhdGUuc2gKZWNobyAiWyQoZGF0ZSAnKyVZLSVtLSVkICVIOiVNOiVTJyldIHByZXVwZGF0ZS5zaCBkZXBsb3llZCBzdWNjZXNzZnVsbHkgb24gJChob3N0bmFtZSAtZikiCg== | base64 -d | bash"

Expected output (one line per node):

[####-##-## ##:##:##] preupdate.sh deployed successfully on <node-fqdn>

After deploying the script, verify it is present on all nodes by running the following command from any node:

vracli cluster exec -- bash -c "ls -la /var/vmware/prelude/upgrade/run/preupdate.sh && echo OK on \$(current_node)"

Proceed with the patch upgrade as normal.

Additional Information

Logs from the preupdate.sh script execution can be found on each node at /var/log/bootstrap/preupdate-patch.log.