VCF Automation 9.0.x Upgrade fails with Error Code LCMVMSP10036 and package deployment vcfa-bundle stuck in Progressing phase

Products

• VCF Automation

Issue/Introduction

• When you attempt to apply VCF Automation product patch via VCF Fleet Management, the task fails with the following error message:

  Error Code: LCMVMSP10036
  Applying product patch failed.
  -- PACKAGE DEPLOYMENT [prelude/vcfa-bundle] STATUS:
  Progressing -- package depbyment is in progress -- BUNDLE
  [prelude/vcfa-bundle] STATUS: Failed -- failed to apply the
  PackageDeployment: change already in progress
  deployment watch failed: context deadline exceeded -- HR
  [prelude/vksm-stack] READY: False (DependencyNotReady)

• When you check the cluster, the vcfa-bundle package deployment is stuck in a Progressing phase (STATUS : package deployment is in progress).

  

• kubectl get pods -A commands shows multiple vcfa-server-manager pod in evicted state except one in running state.

• Version of vcfa-bundle and vmsp-platform package deployment using below command shows vcfa-bundle is upgraded whereas vmsp-platform did not.

  kubectl get pd vmsp-platform -n vmsp-platform -o json | jq -r .metadata.labels

  kubectl get pd vcfa-bundle -n prelude -o json | jq -r .metadata.labels

• Helm release status using the command kubectl get hr -n prelude shows vksm-stack ready status as False and vmsp-prelude-deployer as Unknown. 

Environment

VCF Automation 9.0.x

Resolution

Ensure a valid SFTP backup of VCF Automation 9.0.x is available before performing the below steps :

1. SSH into VCF Automation appliance using vmware-system-user and switch to root user :

   sudo su
   export KUBECONFIG=/etc/kubernetes/admin.conf

2. Identify the vcfa-server-manager pods in an Evicted state by running the following command:

   kubectl get pods -n prelude
   

3. Delete each of the evicted vcfa-server-manager pods using the below command:

   kubectl delete pod -n prelude vcfa-server-manager-##### 

4. Confirm that no pods are in an error state and verify that the package deployment has completed successfully:

   kubectl get packagedeployments -A
   

   Note: The STATUS for vcfa-bundle should now read successful package deployment.

5. Return to the Fleet Management UI and retry the failed VCF Automation patch task.