VCF Automation deployment fails with LCMVSPHERECONFIG1000095 on Fleet Manager

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When you attempt to deploy or redeploy VMware VCF Automation, the task fails during the initialization of the bootstrap machine. This issue typically occurs after a VCF 9.0.2 patch is applied to the Fleet Manager appliance. You will see the following error in the VCF UI: Failed to create services platform cluster. Refer to /var/log/vrlcm/vmsp_bootstrap_xxxxx.log for more details.

Inside the /var/log/vrlcm/vmsp_bootstrap_xxxx.log, the following Docker and iptables errors are present:

docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint package-registry... 
iptables v1.8.9 (nf_tables): RULE_APPEND failed (No such file or directory): rule

Environment

• Product: VMware Cloud Foundation (VCF) 9.0.2
• Component: Fleet Manager (Lifecycle Management)
• Service: VMware VCF Automation

Cause

The issue is caused by a kernel mismatch on the Fleet Manager appliance. The VCF 9.0.2 patch installs a newer kernel and updated iptables packages. If the appliance is not rebooted after the patch, the new iptables binaries will fail to function correctly because they require kernel modules only available in the newer, currently inactive kernel.

Resolution

To resolve this issue, you must ensure the Fleet Manager appliance is running the kernel version that matches its installed packages.

1. Log in to the VCF Fleet Manager appliance via SSH as root.
2. Verify if a kernel mismatch exists by running:
   • Check running kernel: uname -r
   • Check installed kernel: rpm -qa | grep linux-
3. If the versions do not match, reboot the Fleet Manager appliance:

   reboot

4. Once the appliance is back online, log in to the VCF UI.
5. Navigate to the failed deployment task and click Retry.

Additional Information

Restarting the docker or containerd services is insufficient to resolve this issue as the underlying kernel modules required for networking remain unavailable until the system is rebooted.