Error: "Cannot access the vCenter, vmware-sps vmware-vpxd not starting" for the Standalone vCenter

Products

• VMware vCenter Server

Issue/Introduction

• Cannot access the vCenter UI
• vmware-sps and vmware-vpxd stuck in StartPending when running below command

  service-control --status --all

• Certificates are not expired, checked with vCert - Scripted vCenter expired certificate replacement
• vpxd.cfg is not corrupted, checked with Validating corrupt VCSA (vpxd.cfg)
• Partitions are not full 

  df -h
  Filesystem                                   Size  Used Avail Use% Mounted on
  devtmpfs                                     4.0M     0  4.0M   0% /dev
  tmpfs                                         11G  640K   11G   1% /dev/shm
  tmpfs                                        4.2G  9.3M  4.1G   1% /run
  tmpfs                                        4.0M     0  4.0M   0% /sys/fs/cgroup
  /dev/mapper/vg_root_0-lv_root_0               47G   16G   30G  35% /
  tmpfs                                         11G  4.4M   11G   1% /tmp
  /dev/mapper/imagebuilder_vg-imagebuilder     9.8G   36K  9.3G   1% /storage/imagebuilder
  /dev/mapper/netdump_vg-netdump               982M   24K  915M   1% /storage/netdump
  /dev/mapper/core_vg-core                      49G   47M   47G   1% /storage/core
  /dev/mapper/archive_vg-archive                49G  5.4G   42G  12% /storage/archive
  /dev/sda3                                    488M   37M  415M   9% /boot
  /dev/mapper/dblog_vg-dblog                    15G  705M   14G   5% /storage/dblog
  /dev/mapper/vtsdb_vg-vtsdb                    25G   35M   24G   1% /storage/vtsdb
  /dev/mapper/autodeploy_vg-autodeploy         9.8G   40K  9.3G   1% /storage/autodeploy
  /dev/mapper/updatemgr_vg-updatemgr            98G  2.7G   91G   3% /storage/updatemgr
  /dev/sda2                                     10M  2.0M  8.1M  20% /boot/efi
  /dev/mapper/lifecycle_vg-lifecycle            98G  3.8G   90G   5% /storage/lifecycle
  /dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot  196G   28K  186G   1% /storage/lvm_snapshot
  /dev/mapper/vtsdblog_vg-vtsdblog              15G   33M   14G   1% /storage/vtsdblog
  /dev/mapper/log_vg-log                       9.8G  1.7G  7.6G  19% /storage/log
  /dev/mapper/db_vg-db                         9.8G  2.5G  6.8G  28% /storage/db
  /dev/mapper/seat_vg-seat                      25G  225M   23G   1% /storage/seat
  tmpfs                                        1.0M     0  1.0M   0% /var/spool/snmp
  overlay                                       47G   16G   30G  35% /storage/containers/vc-ws1a-broker/######################################################/rootfs

Environment

vSphere vCenter Server 8.X

vSphere vCenter Server 7.X

Cause

• Domain name has been changed for the Standalone vCenter
• Confirming below messages in /var/log/vmware/vpxd/vpxd.log 

  warning vpxd[06976] [Originator@6876 sub=UserDirectorySso] Could not normalize domain : N3Vpx6Common3Sso23DomainNotFoundExceptionE(No Domain found with ID: )
  
  info vpxd[06976] [Originator@6876 sub=SsoWrapper.DomainIdNormalizer] Domain cache current state:
  --> systemDomainName:
  --> sso-to-vpxd-map:
  --> vpxd-to-sso-map:
  --> federated-domains:
  -->
  info vpxd[06976] [Originator@6876 sub=SsoWrapper.DomainIdNormalizer] Domain cache updated state:
  --> systemDomainName: vsphere.local
  --> sso-to-vpxd-map:
  -->   "VSPHERE.LOCAL": "VSPHERE.LOCAL"
  --> vpxd-to-sso-map:
  -->   "VSPHERE.LOCAL": "vsphere.local"
  --> federated-domains:
  -->
  
  
  

Resolution

1. Take appropriate snapshots of the vCenter. Online snapshots if this is a standalone vCenter (as in our example) and offline snapshot is there is linked mode enabled. Read more at Snapshot Best practices for vCenter Server Virtual Machines
2. Run below command to execute domain repoint 

   cmsso-util domain-repoint -m execute --src-emb-admin Administrator  --dest-domain-name vsphere.local

Additional Information

• Note: 

  AD identity source configuration must be recreated after repoint 
   
  All custom vsphere.local user and group information, like usernames and passwords must be recreated after repoint 
   
  Global Permissions must be recreated after the repoint

• For more information about domain repointing visit Repoint vCenter Server to Another vCenter Server in a Different Domain