Upgrade Failure of VCF Automation and VMware Identity Broker from 9.0.0.0 to 9.0.1.0 with Error "LCMVMSP10037"

Products

• VCF Automation

Issue/Introduction

During the upgrade of VMware Cloud Foundation (VCF) Automation or VMware Identity Broker from version 9.0.0.0 to 9.0.1.0, the upgrade fails on Fleet Management with error LCMVMSP10037.

In the /var/log/vrlcm/vmware_vrlcm.log, the following error can be seen:

INFO vrlcm[756715] [http-nio-8080-exec-6] [c.v.v.l.r.c.RequestController] – Retry error cause data : [ {
"messageId" : "LCMVMSP10037",
"message" : "Applying patch on the application platform failed.",
"retry" : true,
"exceptionMessage" : "Product deployment failed.",
"exceptionStackTrace" : "com.vmware.vrealize.lcm.vmsp.common.exception.VmspPatchingException: No failed pods found.\n\nNo failed pods found.\n\n\tat com.vmware.vrealize.lcm.vmsp.plugin.tasks.VmspPatchApplicationPlatformTask.execute(VmspPatchApplicationPlatformTask.java:81)\n\tat com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
"localizedMessageId" : "LCM_VMSP_PKG_PUSH_ERROR_0001",
"parameters" : [ "No failed pods found.\n\nNo failed pods found.\n" ],

Additionally, when checking the bundle status in VCF Automation or VMware Identity Broker, the following failure is observed:

root@ [ ~ ]# kubectl get bundle -A
NAMESPACE       NAME   AGE   PHASE    STATUS
vmsp-platform   vmsp   66m   Failed   failed to import /tmp/vmsp-pkg3021982879/vmsp-photon-kube-v1.32.0/ovas/cap-poi/vcf-services-platform-template-9.0.1.0.24940697.ova: network '/datacenter/network/network-name' not found

OR 

root [~]# kubectl get bundle -A
NAMESPACE       NAME   AGE    PHASE    STATUS
vmsp-platform   vmsp   5m1s   Failed   failed to get the datastore: datastore '/datacenter/datastore/datastore-name' not found

Environment

VCF Automation 9.x
VMware Identity Broker 9.x

Cause

The upgrade fails because the vSphere network or datastore configured in the vmsp-platform Provider Definition (PD) does not exist or has been renamed in vCenter. As a result, the OVA import fails with the error “network not found” or "datastore not found".

Resolution

Step 1: Launch a temporary debug pod:

Run the following command to start a debug session in the vmsp-platform namespace:

kubectl run -n vmsp-platform --restart=Never --rm -i --tty debug --image=registry.vmsp-platform.svc.cluster.local:5000/images/debug:v1.0.7 --command -- bash

Note: This command launches a temporary debugging pod with an interactive shell for network verification or connectivity testing.

Step 2: Authenticate to vCenter using govc

export GOVC_URL=<VCfqdn>
export GOVC_USERNAME=<vCenter-username>
export GOVC_PASSWORD=<vCenter-password>
export GOVC_INSECURE=1

List available network objects:

govc find -type n .

List available datastore objects:

govc datastore.info

Example output:

/Datacenter/network/Management Networks/network-name
/Datacenter/network/test/network-name

Check if the expected network path exists or if it has changed.

Name:        <datastore-name
  Path:      /datacenter/datastore/foldername/datastore-name
  Type:      VMFS

Step 3: Exit from temporary debug pod:

root@ [ ~ ]# exit

Step 4: Validate PD configuration

SSH into any cluster node.

Set the kubeconfig path:

export KUBECONFIG=/etc/kubernetes/admin.conf

Check PD status:

kubectl get pd -n vmsp-platform vmsp-platform

Example output: 

root@ [ ~ ]# kubectl get pd -n vmsp-platform vmsp-platform

NAME            AGE    PHASE         STATUS

vmsp-platform   120d    successfull    successfull package deployment

 

If not Successful, stop here and engage Broadcom Technical Support.

 

Edit PD configuration:

 

kubectl edit pd -n vmsp-platform vmsp-platform   >> then find the network name under spec->values->provider->vsphere->network and change it the right name. 

 

provider:

      vsphere:

     network: /Datacenter/network/Management Networks/network-name

For datastore: 

provider:

      vsphere:

     datastore: /Datacenter/datastore/folder/datastore-name

Monitor PD until it reaches a successful state:

 

kubectl get pd -n vmsp-platform vmsp-platform

 

Example output: 

 

NAME            AGE    PHASE        STATUS

vmsp-platform   120d   Successful   successful package deployment

 

Note: It might take 30-60 minutes for PD status to become Successful.

Step 5: Retry the upgrade

Once PD shows Successful, restart the upgrade from Fleet Management.

Additional Information

The error LCMVMSP10037 indicates a patching failure caused by missing or invalid vSphere configuration.

The govc find command helps verify the actual vSphere network path.

Always ensure vCenter inventory objects referenced in PD (network, datastore, cluster) are not renamed or removed before an upgrade