Upgrading Identity Broker and VCF Automation from 9.0.1 to 9.0.2 fails with error: LCMVMSP10010

Products

• VCF Operations

Issue/Introduction

• When upgrading Identity Broker and VCF Automation from version 9.0.1 to 9.0.2 within VMware Cloud Foundation (VCF) Operations Fleet Management, the process fails during the backup phase.
• The following error message is displayed in the LCM log or UI:

Error Code: LCMVMSP10010

Failed to take backup of the application platform.
Exception occurred during backup of the application platform. Exception message: Failed to fetch authorization token, Attempts exceed after retried for 10 times. Please ensure the instance pods are available/healthy. Response: {"error":"error: error loading config file \"/dev/stdin\": couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }\n","output":"","exitCode":1}.

 

Environment

VCF Identity Broker 9.0.x

VCF Automation 9.0.x

VCF Operations Fleet Management (LCM)

Cause

The failure is caused by the Fleet Management component's inability to retrieve the correct version/kind schema from the configuration file at /dev/stdin.

Resolution

To resolve this issue, you must clear the appliance cache by performing a graceful reboot of the Fleet Management appliance. 

 

1. VIDB Appliance:
Before rebooting Fleet Management, verify the status of the Identity Broker (VIDB) appliance:

• Pod Health: Ensure that all Kubernetes pods related to the instance are available and in a healthy/running state.

• • SSH to the VIDB appliance as vmware-system-user.
  • Change the user to root:

    sudo su root

  • Export kubeconfig details (to use kubectl commands):

    export KUBECONFIG=/etc/kubernetes/admin.conf

  • List all namespaces and check pod status:

    kubectl get pods -A

  • Check the Docker service status:

    service docker status

• Disk Space: Verify that there are no disk space capacity issues on the appliance, as a full disk can cause similar symptoms.
  • • Check disk space using the following command:

      df -h

• Perform a graceful reboot of the Fleet Management appliance VM
• After the appliance is back online, wait few minutes for all internal services to fully initialize on the Fleet Management Appliance
• Navigate back to the Fleet Management UI and re-initiate the Identity Broker upgrade from 9.0.1 to 9.0.2. The backup and update process should now complete successfully.

2. VCF Automation:

Before proceeding with the Fleet Management appliance VM reboot, ensure any stalled or failed upgrade tasks are cleared:

• Follow the steps outlined in the KB# 415510 to cancel the previously failed or stalled automation upgrade tasks from Fleet Management UI. 
• Perform a graceful reboot of the Fleet Management appliance VM.
• After the appliance is back online, wait few minutes for all internal services to fully initialize on the Fleet Management Appliance.
• Navigate back to the Fleet Management UI and re-initiate the Automation upgrade from 9.0.1 to 9.0.2. The backup and update process should now complete successfully.