vCenter Server UI login fails with "no healthy upstream" error due to vmdir database(data.mdb) corruption

Products

• VMware vCenter Server

Issue/Introduction

• Logging in to the vCenter UI fails with a "no healthy upstream" error
• Starting all the vCenter services(service-control --status --all) will fail with the following error:
  
  
  ERROR service-control Error executing start on service vmdird. Details {
      "resolution": null,
      "detail": [
          {
              "args": [
                  "vmdird"
              ],
              "id": "install.ciscommon.service.failstart",
              "localized": "An error occurred while starting service 'vmdird'",
              "translatable": "An error occurred while starting service '%(0)s'"
          }
      ],
      "componentKey": null,
      "problemId": null
  }
  ERROR service-control Service-control failed. Error {
      "resolution": null,
      "detail": [
          {
              "args": [
                  "vmdird"
              ],
              "id": "install.ciscommon.service.failstart",
              "localized": "An error occurred while starting service 'vmdird'",
              "translatable": "An error occurred while starting service '%(0)s'"
          }
      ],
      "componentKey": null,
      "problemId": null
  }

• The /var/log/vmware/vmdird/vmdird.log file from the vCenter shows the following error:
  
  YYYY-MM-DDTHH:MM:SSZ:t@###########:ERROR: LoadReplicationAgreements: InternalSearch for Replication Agreements failed. Error code: 9703, Error string: DNToEID (9703)((MDB_NOTFOUND: No matching key/data pair found)(cn=<vCenter PNID>,cn=servers,cn=default-first-site,cn=sites,cn=configuration,dc=vsphere,dc=local))

Cause

The error code 9703 and the string MDB_NOTFOUND in the /var/log/vmware/vmdird/vmdird.log indicates that the directory service is unable to locate essential entries within the database file. This specific error signature is the primary indicator of a corrupted or inconsistent data.mdb file. This file serves as the database for the VMware directory service, and its corruption prevents the service from retrieving replication agreements or server configurations

Resolution

1. Restore the /storage/db/vmware-vmdir/data.mdb file using the backup located at /storage/db/vmware-vmdir/snapshot/

mv /storage/db/vmware-vmdir/data.mdb /var/core/
cp /storage/db/vmware-vmdir/snapshot/data.mdb /storage/db/vmware-vmdir/
   
   
2. Start the vmdird service
   
   service-control --start vmdird
   
   
3. Repoint the vCenter to its current domain to ensure that all service registrations are correctly updated
   
   cmsso-util domain-repoint -m execute --src-emb-admin Administrator --dest-domain-name destination_PSC_domain
   
   For more information, refer: Domain Repoint