Changing the vCenter FQDN fails due to large vmdir database size

Products

VMware vCenter Server

Issue/Introduction

vCenter hostname change fails at 75% with the error message:
```
Failed to reset machine account
```

In /var/log/vmware/applmgmt/pnid_change.log there are entries similar to the below:

YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():81 - INFO - Running command: /usr/bin/service-control --start vmafdd vmdird vmca
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():99 - INFO - Command exited with exitcode : 0
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():109 - INFO - Done running command
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():81 - INFO - Running command: /usr/lib/vmware-vmdir/bin/vdcresetMachineActCred -u administrator
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():99 - INFO - Command exited with exitcode : 53
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():101 - WARNING - Command failed with following error:
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():102 - WARNING - b'password: \n'
YYYY-MM-DDTHH:MM:SS - pnid_utils-run_command():109 - INFO - Done running command
YYYY-MM-DDTHH:MM:SS - pnid_utils-restart_services():1598 - ERROR - password:
YYYY-MM-DDTHH:MM:SS - pnid_utils-update_task_status():1444 - INFO - Task : Failed to reset machine account.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The cause of this issue is the large size of the vmdir database.

Due to the large size, the database snapshot creation takes longer and vmdir state will be in read-only till the snapshot creation is complete. Resetting the machine account during this time will fail with the above error.

Resolution

To confirm if the issue matches this KB, start with validating the state of vmdir at the time the command to reset the machine account was issued during the PNID change workflow. For example:

In the pnid_change.log we can see that the vmdird service start issued:

YYYY-MM-DDT03:00:17 - pnid_utils-run_command():81 - INFO - Running command: /usr/bin/service-control --start vmafdd vmdird vmca

In the /var/log/vmware/vmdird/vmdird-syslog.log we can see the vmdird state turned to NORMAL at 03:00:28:

YYYY-MM-DDTHH:MM:SS info vmdird  t@140075752752960: Lotus Vmdird: starting..
YYYY-MM-DDTHH:MM:SS info vmdird  t@140053825177344: Taking snapshot, setting backend state to MDB_RDONLY
YYYY-MM-DDTHH:MM:SS info vmdird  t@140053825177344: _VmDirCpMdbFile: making database snapshot with file size 2072Mb; will take approximate 41 seconds; 0 updates occurred since last snapshot.
YYYY-MM-DDT03:00:28 info vmdird  t@140053825177344: Backend state set to NORMAL

The machine account reset failed at 03:00:23 while vmdird was still in read-only mode:

YYYY-MM-DDT03:00:23 - pnid_utils-update_task_status():1444 - INFO - Task : Failed to reset machine account.

Once this have been confirmed, apply the following steps to work around the issue, by temporarily disabling database snapshots during the vmdird service startup

Ensure that a fresh offline snapshot of the vCenter Server Appliance exists. If the VCSA is a member of an Enhanced Linked Mode replication setup, be aware that offline snapshots of all ELM members are required
Open an SSH connection to the VCSA and login with root

Run the below command to disable snapshots of the vmdird database:

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir\Parameters]' "CopyDbIntervalInSec" "0"

Restart the vmdird service:
```
service-control --restart vmdird
```
Follow the usual process to change the vCenter hostname/PNID. For more information please refer to Reconfigure the Primary Network Identifier

After the hostname has been successfully changed, reenable automatic snapshots:

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir\Parameters]' "CopyDbIntervalInSec" "60"

Restart the vmdird service.
```
service-control --restart vmdird
```

Additional Information

If you receive a similar error during the PNID change, but vmdird is confirmed to have been in normal mode during the failure, please refer PNID Change failing with " Failed to reset machine account " for other causes of this issue.