VCF Operations for Network upgrade to 9.1 fails at 'Inventory Sync Post Upgrade' due to Log Management being in an unknown status

Products

• VCF Operations

Issue/Introduction

• VCF Operations for Network upgrade to version 9.1 fails at 'Inventory Sync Post Upgrade' with the following error:
  
  Could not complete Fleet lifecycle task with ID '######-####-####-####-#######'. Check Fleet Lifecycle and SDDC lifecycle logs for details.
Step post_upgrade_inventory_sync failed. Reference Code: ########. Check logs for details.


• On Fleet Lifecycle Management (VCF_FLEET_LCM), in /services-logs/vcf-fleet-lcm/<pod_name>/vcf-fleet-build-service-fleetbuild-####-##/fleetbuild/console.log:

YYYY-MM-DDTHH:MM:SS WARN  Fleet Build ## [lcm@#### scPath="ops" thread="system-task-worker-##" logger="c.b.v.l.f.b.o.s.OpsPlatformServiceImpl" method="lambda$retryFailedTask$6:L"] [, taskId=######-####-####-####-#######] - Component registration task with ID ######-####-####-####-####### encountered an error Error occurred while creating Service Account in VCF Operations for Logs in Ops Platform with message: java.net.ConnectException: Connection refused

"Error occurred while creating Service Account in VCF Operations for Logs","messageId":"LCMVRLICONFIG40116","parameters":[],"properties":{},"retry":true}]}
  
  
• Under Build -> Lifecycle -> VCF Management -> Components -> Log Management, the cluster is in 'unknown' status.
  
  
• The Log Management UI version 9.0.2 displays the cluster in a degraded state.

Environment

VMware Cloud Foundation 9.1.x

Cause

• The Log Management cluster running on version 9.0.2 crashes due to a known issue where the cluster runs out of memory. Refer: Primary node crashes intermittently - VCF Operations for Logs 9.0.x
• The Inventory sync post upgrade failure for VCF Operations for Networks occurs because VCF Operations for Logs is in an 'unknown' state and unreachable.
• By design, the OPS Service Registry automatically attempts to sync dependent components after an upgrade. Because the Logs cluster is unresponsive, the system throws a "Connection refused" error when trying to create a service account, which causes the post-upgrade inventory sync to fail.

Resolution

1. Patch the Log Management Cluster to version 9.0.2.0200 to resolve the outofmemory error. Refer: Upgrade VMware Aria Operations for Logs
2. Once the cluster is in an active state under Build -> Lifecycle -> VCF Management -> Components -> Log Management, restart the failed 'Inventory Sync Post Upgrade' task on VCF Operations.

Additional Information

Primary node crashes intermittently - VCF Operations for Logs 9.0.x

Upgrade VMware Aria Operations for Logs