Deployment of external vIDB nodes through Fleet Manager fails with error code LCMVSPHERECONFIG1000095.

Products

• VCF Operations

Issue/Introduction

• While deploying external vIDB nodes via Fleet Manager, the deployment fails with error code LCMVSPHERECONFIG1000095.
  
  
• Logs on Fleet Manager located at /var/log/vrlcm/vmware_vrlcm.log shows entries similar to:

YYYY-MM-DDThh:mm:ss INFO vrlcm[319498] [pool-3-thread-23] [c.v.v.l.v.p.u.VMSPUtil]-- VMSP Provisioning process exited with error code 1
YYYY-MM-DDThh:mm:ss INFO vrlcm[319498] [pool-3-thread-23] [c.v.v.l.v.p.u.VMSPUtil]-- Fetching errors
YYYY-MM-DDThh:mm:ss INFO vrlcm[319498] [pool-3-thread-23] [c.v.v.l.v.p.u.VMSPUtil]-- Capture errors exited with code:: 2
YYYY-MM-DDThh:mm:ss ERROR vrlcm[319498] [pool-3-thread-23] [c.v.v.l.v.p.t.BootstrapVMSPTask]-- Failed to create services platform cluster. Refer to /var/log/vrlcm/vmsp_bootstrap_xxxxx.log for more details.
.
.
YYYY-MM-DDThh:mm:ss INFO vrlcm[319498] [pool-3-thread-23] [c.v.v.l.p.a.s.Task]-- Injecting task failure event. Error Code : 'LCMVSPHERECONFIG1000095', Retry : 'true', Causing Properties : '{ CAUSE :: ipPool === primaryVip === dns === network === vmwareSystemUserPassword YXYXYXYX vcPassword YXYXYXYX netmask === cluster === vcUsername === datastore === vCenterHost === gateway === vmwareSystemUserPassword YXYXYXYX deployOption === resourcePool === folderName === internalClusterCidr === vmNamePrefix === additionalVips === domain ===}'
java.lang.Exception: Failed to create services platform cluster. Refer to /var/log/vrlcm/vmsp_bootstrap_xxxxx.log for more details.

• Additionally, logs on Fleet Manager located at /var/log/vrlcm/vmsp_bootstrap_xxxxx.log shows entries similar to:
  
  E0430 hh:mm:ss 2436798 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<VIDB-VIP>:6443/api?timeout=32s\": dial tcp <VIDB-IP>:6443: i/o timeout"
E0430 hh:mm:ss 2436798 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<VIDB-VIP>:6443/api?timeout=32s\": dial tcp <VIDB-IP>:6443: i/o timeout"
E0430 hh:mm:ss 2436798 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<VIDB-VIP>:6443/api?timeout=32s\": dial tcp <VIDB-IP>:6443: i/o timeout"
E0430 hh:mm:ss 2436798 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<VIDB-VIP>:6443/api?timeout=32s\": dial tcp <VIDB-IP>:6443: i/o timeout"
E0430 hh:mm:ss 2436798 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://<VIDB-VIP>:6443/api?timeout=32s\": dial tcp <VIDB-IP>:6443: i/o timeout"
Unable to connect to the server: dial tcp <VIDB-VIP>:6443: i/o timeout

Environment

VCF Operations Fleet Manager 9.x
VCF Identity Broker 9.x

Cause

The issue occurred because port 6443 was blocked between the VCF Operations Fleet Manager and the vIDB VIP. As a result, Fleet Manager was unable to establish communication with the Control Plane API server during the VMSP bootstrap process, leading to the deployment failure.

Resolution

Ensure that all required ports are open and accessible between the VCF Operations Fleet Manager and the VCF Identity Broker environment.

Refer to the official Broadcom Ports and Protocols documentation below for the complete connectivity requirements:

https://ports.broadcom.com/home/VMware-Cloud-Foundation-Operations-Fleet-Manager