VCO Pods Fail to Start After Upgrade to Aria Automation 8.18.1 Patch 2
Article ID: 411141
Updated On:
Products
Issue/Introduction
VCO Pod will not start when VC plugin is enabled, when we disable VC plugin the vco pod starts normally
Environment
Aria Automation 8.18.1 Patch 2
Cause
The environment contains a large number of configuration elements with embedded virtual machines. During startup, the Orchestrator’s content indexer attempts to resolve the details of these VMs.
Below events under /service-logs/prelude/vco-app/file-logs/vco-app-server.log indicate that Orchestrator is attempting to resolve deleted or stale VM objects during startup.
INFO vco ... Loading session 'HostSessionKey [hostname=####, username=####, session: SdkSession]'.
ERROR vco ... Exception during executing finder for type 'VirtualMachine' and id '####'.
com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.VirtualMachine:vm-######' has already been deleted or has not been completely created
In Patch 2, the vSphere plugin cache TTL (time-to-live) was reduced (from 10 minutes to 5 minutes) to ensure fresher data synchronization with vCenter. This change causes repeated fetch operations during the initial indexing process, which leads to vCO pod startup failures.
In the recent versions of the vCenter plugin 8.2.1, a retry mechanism was introduced. Depending on the customer’s custom content and inventory items, this mechanism may interfere with the normal startup of the Orchestrator server. In earlier versions of the vCenter plugin, this mechanism was not present, and therefore no retry attempts were made to search for missing or stale inventory items.
Based on the logs, vRO was attempting to locate virtual machines in vCenter that no longer exist. Since these objects could not be reindexed, the vco pod failed to start.
Resolution
SSH into one of the vRA node and run the below command to disable the retry mechanism.
vracli vro properties set -k "o11n.plugin.vc.client-utils.retries" -v "-1"
Feedback
