Upgrade of VCF 9.0 to 9.1 fails with error "Import VCF Operations in Fleet Lifecycle Failed"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Upgrading VCF 9.0 to VCF 9.1 fails with the following error:
  
  Import VCF Operations in Fleet Lifecycle Failed


• The VCF Operations log /storage/log/vcops/log/component-manager-plugin-*.log contains errors indicating a failure to connect to VCF Automation:
  
  [com.vmware.vrealize.lcm.vcfa.plugin.tasks.PushCapabilitiesToVcfaTask.lambda$pushCapabilities$0] - Error occurred while pushing capabilities to VCF Automation
com.vmware.vrealize.lcm.vcfa.common.exception.UnableToGetVcfaTokenException: Connection refused
  
  
• Checking the VCF Automation virtual machine console in vCenter server shows CPU stuck or an unresponsive state.
  
  watchdog: BUG: soft lockup - CPU## stuck for #s! [swapper/#:#]

Environment

• VMware Cloud Foundation 9.0

• VMware Cloud Foundation 9.1

Cause

A crash or unresponsiveness of VCF Automation prevents VCF Operations from establishing a connection, causing the registration of VCF Operations in Fleet Lifecycle to fail during the upgrade.

Resolution

• Verify that sufficient CPU resources are allocated to the VCF Automation virtual machine.

• Log in to vCenter server using vCenter server SSO administrator credentials and restart the VCF Automation virtual machine.

• Run the following command to verify that VCF Automation service pods return to normal status:
  
  kubectl get pods -A | grep prelude

• Retry the Fleet lifecycle upgrade task.