VCO Pods Fail to Start After Upgrade to Aria Automation 8.18.1 Patch 2

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Aria automation is upgraded to 8.18.1 Patch 2 after setting following properties and disabling VC plugin before upgrade which resolved the upgrade failure issues due to vco pod initialization problem.

vracli vro properties set -k com.vmware.vmo.plugin.vi4.cache.live.objects.expirationSeconds -v 720
vracli vro properties set -k com.vmware.vmo.plugin.vi4.cache.main.expirationSeconds -v 720

After the successful upgrade to Aria Automation 8.18.1 Patch 2, the vco pods fails to start when the vSphere (VC) plugin is enabled. 

Environment

Aria Automation 8.18.1 Patch 2

Cause

The environment contains a large number of configuration elements with embedded virtual machines. During startup, the Orchestrator’s content indexer attempts to resolve the details of these VMs. 

Below events under /service-logs/prelude/vco-app/file-logs/vco-app-server.log indicate that Orchestrator is attempting to resolve deleted or stale VM objects during startup.

INFO  vco ... Loading session 'HostSessionKey [hostname=####, username=####, session: SdkSession]'.
ERROR vco ... Exception during executing finder for type 'VirtualMachine' and id '####'.
com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.VirtualMachine:vm-######' has already been deleted or has not been completely created

In Patch 2, the vSphere plugin cache TTL (time-to-live) was reduced (from 10 minutes to 5 minutes) to ensure fresher data synchronization with vCenter. This change causes repeated fetch operations during the initial indexing process, which leads to vCO pod startup failures.

Resolution

Option 1:

Increase the startup probe’s failurethreshold for the vco-server-app container within the vco-app pod

• Run the following command to edit the deployment:

  kubectl -n prelude edit deployment vco-app

• Locate the startupProbe section and update: failureThreshold from 50 to 300
• Enable the vSphere plugin

Option 2:

Before proceeding with the upgrade, review the custom content that references deleted virtual machines and remove those entries. Since the VMs are already deleted, removing the references will prevent Orchestrator from attempting to index them.