Fleet Management node disconnected in VCF Operations after upgrading to version 9.0.2

Products

• VCF Operations

Issue/Introduction

After upgrading Fleet Management to version 9.0.2, the node appears as "disconnected" within the VCF Operations Admin UI (System Status > Fleet Management page). Manual attempts to reconnect the appliance fail with the error: "Error connection Fleet management node, try again."

• No new log entries are generated in /var/log/vrlcm/vmware_vrlcm.log on the Fleet Management appliance post-upgrade.

• Error patterns are observed in /storage/vcops/log/casa/casa.log on the VCF Operations node.

  ERROR casa ##### [ops@#### threadId="###" threadName="ajp-nio-###.0.0.1-#####-exec-9" requestId="########"] [com.vmware.vcops.casa.fleetmanagement.FleetManagementNodeService.handleNodeAddition:###] - Exception during node addition:
  com.vmware.vcops.casa.exception.CasaLocalizableException: CasaLocalizableException: key=fleet.node.management.already.registered; args=; cause=
          at com.vmware.vcops.casa.fleetmanagement.FleetManagementNodeService.handleNodeAddition(FleetManagementNodeService.java:###) ~[classes/:?]
          at com.vmware.vcops.casa.fleetmanagement.FleetManagementNodeController.nodeAddition(FleetManagementNodeController.java:##) ~[classes/:?]
  at com.vmware.vcops.casa.fleetmanagement.FleetManagementNodeController$$FastClassBySpringCGLIB$$bb472786.invoke(<generated>) ~[classes/:?]
          at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:###) ~[spring-core-#.#.##.jar:#.#.##]
          at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:###) ~[spring-aop-#.#.##.jar:#.#.##]

• Upgrade failure signatures are present in /var/log/vmware/capengine/cap-non-lvm-update/workflow.log

Failed with status of 1003 while installing version 9.0.2.0
VM version is still 9.0.0.0
:exit status 1
####/##/## ##:##:##.###### non_lvm_update_post_update_script_plugin.go:##: post install extension script failed exit status 1
####/##/## ##:##:##.###### workflow_manager.go:###: Task ext-post-install failed. Error: exit status 1
####/##/## ##:##:##.###### workflow_manager.go:###: Updating instance status to Failed
####/##/## ##:##:##.###### parameter_logger.go:##: Execution parameters

Environment

VCF Operations 9.x

VCF Fleet Manager 9.x

Cause

The Fleet Management appliance was disconnected due to a failed upgrade, which was caused by expired root passwords on both the Fleet Management and VCF Operations appliances

Resolution

To resolve this issue, please follow the steps outlined below:

• Revert the Fleet Management appliance to the snapshot taken prior to the 9.0.2 upgrade attempt.

• Confirm that the passwords for both the VCF Fleet Management appliance and VCF Operations are in an Active state. This can be verified under:
  VCF Operations > Fleet Management > Passwords > VCF Management page.

• Initiate an inventory synchronization by navigating to:
  VCF Operations > Fleet Management > Lifecycle > Components > Select Operations > Trigger Inventory Sync,
  and ensure the process completes successfully.

• After completing the above steps, retry the Fleet Management patching to version 9.0.2, following the instructions provided in the “Patch VCF Operations Fleet Management” documentation.