VCO pod crashing continuously post upgrade to Aria orchestrator 8.18.1

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Kubectl get pods -n prelude shows lot restarts for VCO pod
• The UI reports error 500 ok every time VCO pod crashes

     Log location:/services-logs/prelude/vco-pods/file-logs

         YYYY-MM-DDT##:##:## WARN vco [host='vco-app-123456789' thread='pool-123456-thread-1' user='' org='' trace=''] {} com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager - Certificate thumbprint validation appears to be slow (123456 ms).

Environment

 Aria orchestrator 8.18.x

Cause

Any workflow involving vCenter endpoints can lead to a rapid increase in connection objects and the accumulation of hundreds of trusted certificates. Most of the memory usage is attributed to keystores linked to vCenter plugin connection objects

Resolution

Broadcom is aware of this known issue with the Vcenter plugin version seen in 8.18.1 (8.1.1-24147893). 

This fix is available in 8.18.1 update 2 where the Vcenter plugin version will be 8.2.1-24702511. The instructions to apply this patch can be seen at  VMware Aria Automation 8.18.1 Cumulative Update #2 

Technical Insight: The update fixes the issue by not having associated keystore object with each vCenter connection object thus removing the precondition for such an outage to occur.

Workaround:N/A