Upgrade Failure of VCF Automation from 9.0.0.0 to 9.0.1.0 with Error "LCMVMSP10037"

Products

• VCF Automation

Issue/Introduction

During the upgrade of VMware Cloud Foundation (VCF) Automation from version 9.0.0.0 to 9.0.1.0, the upgrade fails on Fleet Management with error LCMVMSP10037.

In the /var/log/vrlcm/vmware_vrlcm.log, the following error can be seen:

2025-10-30T04:30:59.216Z INFO vrlcm[756715] [http-nio-8080-exec-6] [c.v.v.l.r.c.RequestController] – Retry error cause data : [ {
"messageId" : "LCMVMSP10037",
"message" : "Applying patch on the application platform failed.",
"eventId" : "a1faeaba-b5d5-4b00-b7b8-5c1dfd4637ca",
"retry" : true,
"exceptionMessage" : "Product deployment failed.",
"exceptionStackTrace" : "com.vmware.vrealize.lcm.vmsp.common.exception.VmspPatchingException: No failed pods found.\n\nNo failed pods found.\n\n\tat com.vmware.vrealize.lcm.vmsp.plugin.tasks.VmspPatchApplicationPlatformTask.execute(VmspPatchApplicationPlatformTask.java:81)\n\tat com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
"localizedMessageId" : "LCM_VMSP_PKG_PUSH_ERROR_0001",
"parameters" : [ "No failed pods found.\n\nNo failed pods found.\n" ],

Additionally, when checking the bundle status in VCF Automation, the following failure is observed:

root@ [ ~ ]# kubectl get bundle -A
NAMESPACE       NAME   AGE   PHASE    STATUS
vmsp-platform   vmsp   66m   Failed   failed to import /tmp/vmsp-pkg3021982879/vmsp-photon-kube-v1.32.0/ovas/cap-poi/vcf-services-platform-template-9.0.1.0.24940697.ova: network '/datacenter/network/network-name' not found

Environment

VCF Automation 9.x

Cause

The upgrade fails because the vSphere network path configured in the vmsp-platform Provider Definition (PD) does not exist or has been renamed in vCenter. As a result, the OVA import fails with the error “network not found.”

Resolution

Step 1: Launch a temporary debug pod:

Run the following command to start a debug session in the vmsp-platform namespace:

kubectl run -n vmsp-platform --restart=Never --rm -i --tty debug --image=registry.vmsp-platform.svc.cluster.local:5000/images/debug:v1.0.7 --command -- bash

Note: This command launches a temporary debugging pod with an interactive shell for network verification or connectivity testing.

Step 2: Authenticate to vCenter using govc

export GOVC_URL=<VCfqdn>
export GOVC_USERNAME=<vCenter-username>
export GOVC_PASSWORD=<vCenter-password>
export GOVC_INSECURE=1

List available network objects:

govc find -type n .

Example output:

/Datacenter/network/Management Networks/network-name
/Datacenter/network/test/network-name

Check if the expected network path exists or if it has changed.

Step 3: Validate PD configuration

SSH into any cluster node.

Set the kubeconfig path:

export KUBECONFIG=/etc/kubernetes/admin.conf

Check PD status:

kubectl get pd -n vmsp-platform vmsp-platform

Example output: 

root@ [ ~ ]# kubectl get pd -n vmsp-platform vmsp-platform

NAME            AGE    PHASE         STATUS

vmsp-platform   120d    successfull    successfull package deployment

 

If not Successful, stop here and engage Broadcom Technical Support.

 

Edit PD configuration:

 

kubectl edit pd -n vmsp-platform vmsp-platform   >> then find the network name under spec->values->provider->vsphere->network and change it the right name. 

 

provider:

      vsphere:

  network: /Datacenter/network/Management Networks/network-name

 

Monitor PD until it reaches a successful state:

 

kubectl get pd -n vmsp-platform vmsp-platform

 

Example output: 

 

NAME            AGE    PHASE        STATUS

vmsp-platform   120d   Successful   successful package deployment

 

Note: It might take 30-60 minutes for PD status to become Successful.

Step 4: Retry the upgrade

Once PD shows Successful, restart the upgrade from Fleet Management.

Additional Information

The error LCMVMSP10037 indicates a patching failure caused by missing or invalid vSphere configuration.

The govc find command helps verify the actual vSphere network path.

Always ensure vCenter inventory objects referenced in PD (network, datastore, cluster) are not renamed or removed before an upgrade