Troubleshooting failed VCF lcm-bundle-repo removal.

Products

• VMware Cloud Foundation

Issue/Introduction

The lcm-bundle-repo is removed during the drift bundle upgrade in VCF releases 4.5.1 and 5.x for security purposes. 

• The /nfs/vmware/vcf/nfs-mount on the SDDC Manager will still used to store upgrade/install bundles.
• Bundles will now be uploaded to the components via API

In some instances, the drift bundle will succeed without issue but the lcm-bundle-repo will show inaccessible in the vCenter UI and Host Clients.

Environment

VMware Cloud foundation 5.x
VMware Cloud Foundation 4.5.1

Cause

5 potential reasons behind the issue

• Reason#1. VMs or templates present on the NFS share, especially the vCLS VMs.
• Reason#2. VMs have ISOs mounted to the CD/ROM from the lcm-bundle-repo from a prior VCF upgrade. 
• Reason#3. Lockdown mode enabled on the ESXi host 
• Reason#4. A VM is running on a snapshot that contains pointers to the NFS share (ISO mounted before snapshot creation)
• Reason#5. vSphere HA heartbeating is configured on the lcm-bundle-repo

Resolution

Identify why NFS share was not removed during the drift bundle upgrade (using the known factors listed in the section above)

Reason#1 VMs or templates present on the NFS share. Commonly vCLS VMs are incorrectly deployed to local datastores. 

1. Click the "lcm-bundle-repo"

2. Click the "VMs"

3. Check if there are VMs related to this datastore

4. Back to Virtual Machine on Inventory 

5. Select each vCLS VM to check which datastore it running on.

 

Reason#2 VMs have ISOs mounted to the CD/ROM from the lcm-bundle-repo from a prior VCF upgrade. 

1. Click the "lcm-bundle-repo"

2. Click the "VMs"

3. Check if there are VMs related to this datastore

4. If yes, right-click the VM and select "Edit Settings"

5. Check if an ISO is mounted from this storage.

6. If yes, disconnect the CD/DVD, and change it to "Client Device"

 

Reason#3 Lockdown mode enabled on the ESXi host 

1. Select one ESXi server in the inventory.

2. Click "Configure", then click "Security Profile"

3. Check if Lockdown mode is enabled 

 

Reason#4 A VM is running on a snapshot that contains pointers to the NFS share (ISO mounted before snapshot creation)

1. Steps are the same as Reason#2 to locate a VM running on "lcm-bundle-repo"

2. Right-click the VM and select "Snapshot", then click ''Manage Snapshot"

3. Check if this VM running on a snapshot. 

4. If yes, delete this snapshot.

 

Reason#5 vSphere HA heartbeating is configured on the lcm-bundle-repo

1. Select a cluster, then click "Configure"

2. Click "vSphere Availability"

3. Check the "Datastore for Heartbeating" to make sure the "lcm-bundle-repo" is not selected.

After these steps above, now it's safe to manually remove the lcm-bundle-repo from the ESXi host from the vCenter UI.