VMware Identity Broker re-deployment fails with error LCMVSPHERECONFIG1000095 during configuration validation
search cancel

VMware Identity Broker re-deployment fails with error LCMVSPHERECONFIG1000095 during configuration validation

book

Article ID: 436154

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

When attempting to redeploy the VMware Identity Broker (vIDB) appliance in VCF 9.0.x (following an SSO reset or appliance deletion), the deployment fails at the initialization stage. The following error is displayed in the UI:
Failed to create services platform cluster. Refer to /var/log/vrlcm/vmsp_bootstrap_xxxxx.log for more details. yyyy/mm/dd hh:mm:ss role VCF Services Platform exists yyyy/mm/dd hh:mm:ss role VCF Services Platform Admin exists successfully added global permissions for user [email protected] successfully added global permissions for user [email protected] yyyy/mm/dd hh:mm:ss Validated number of IPs provided in IP Pools ERR:INIT0001 - Validating configuration

The vmsp_bootstrap_xxxx.log on the Fleet Manager appliance has entries similar to:
yyyy/mm/dd hh:mm:ss No config file found for the Appliance cluster vcf-mgmt-xxxx-- DEBUG INFO END --E0325 yyyy/mm/dd.xxxxx 6036 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"E0325 yyyy/mm/dd.xxxxx 6036 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"E0325 yyyy/mm/dd.######### memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"E0325 yyyy/mm/dd.######## memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"E0325 ######## 6036 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"error: You must be logged in to the server (the server has asked for the client to provide credentials)

The vmware_vrlcm.log on the Fleet Manager appliance has following entries:
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : IPPOOLS_ADDRESSES -> <IP_range>
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : NODE_PREFIX -> idm-mgmt
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : VSPHERE_RESOURCE_POOL -> ResourcePool:resgroup-##
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : INTERNAL_CLUSTER_CIDR -> #.#.#.#/##
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : DHCP4_ENABLED -> false
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : SSH_PASSWORD YXYXYXYX
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : DOCKER_ROOT_DIR -> /data/docker
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : NAMESERVERS -> <DNS_Server_1_IP_address, DNS_Server_2_IP_address>
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : VSPHERE_NETWORK -> DistributedVirtualPortgroup:dvportgroup-##
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : OPS_MGMT_HOST -> <OPS_FQDN>
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : NTP_SERVERS -> <NTP_Server_FQDN>
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : VSPHERE_PASSWORD YXYXYXYX
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – VMSP Environment Property : PACKAGE_NAME -> ##################
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.u.VMSPUtil] – VMSP Provisioning started
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.u.VMSPUtil] – VMSP Provisioning process exited with error code 9
 INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.u.VMSPUtil] – Fetching errors
INFO vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.u.VMSPUtil] – Capture errors exited with code :: 1
ERROR vrlcm[1249] [pool-3-thread-56] [c.v.v.l.v.p.t.BootstrapVMSPTask] – Failed to create services platform cluster. Refer to /var/log/vrlcm/vmsp_bootstrap_xxxxx.log for more details. yyyy/mm/dd hh:mm:ss role VCF Services Platform exists
yyyy/mm/dd hh:mm:ss role VCF Services Platform Admin exists
successfully added global permissions for user [email protected]
successfully added global permissions for user [email protected]
yyyy/mm/dd hh:mm:ss Validated number of IPs provided in IP Pools
ERR:INIT0001 - Validating configuration

Environment

VCF Operations 9.0.x

Cause

This issue occurs when one or more of the networking fields provided during the deployment wizard are unreachable from the Fleet Manager appliance. The initialization script attempts to validate network connectivity and name resolution; if the endpoint does not respond, the vmsp_bootstrap script triggers a silent timeout before it can proceed to the service platform creation.

Resolution

To identify exactly where the script is hanging, you can enable verbose logging on the Fleet Manager:
(i) Log in to Fleet Manager appliance using root credentials.
(ii) Navigate to /data/vmsp/scripts/
(iii) Open bootstrap.sh script in a text editor: vi bootstrap.sh
(iv) Change the header from set -euo pipefail to set -euox pipefail
(v) Retry the deployment and monitor the logs for the specific command causing the hang.
Example, if DNS server is unreachable, entries will look like this in vmsp_bootstrap_xxxx.log:
+ DNS_RESULT=' ;; communications error to x.x.x.x#53: timed out
;; communications error to x.x.x.x#53: timed out
;; communications error to x.x.x.x#53: timed out


To resolve this issue, verify and correct the endpoint server accessibility identified in the DEBUG level logs from the Fleet Manager:
(i) Review the IP addresses/FQDNs entered during the "Deploy Identity Broker" wizard.
(ii) Log in to the Fleet Manager via SSH using root credentials and test the culprit endpoint server identified using the dig or nslookup command:
# Example test for a specific DNS server
nslookup vmware.com <DNS_SERVER_IP>
(iii) Remove any unreachable or decommissioned servers from the deployment wizard and restart the redeployment process.

Note: Kindly ensure to revert the changes made in bootstrap.sh script to enable DEBUG logging as it will also capture the passwords if the DEBUG logs remain enabled.
Powered by Wolken Software