"An error occurred while starting service 'VMwareDNSService'" - vCenter Server convergence failure during VMAFD firstboot

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter converge process fail during VMAFD firstboot
Running firstboot stage 1 of 4 failed
In /var/log/firstboot/vmafd-firstboot.py_stdout.log, you see entries similar to:

2019-05-07T07:31:50.515Z Running command: ['/opt/likewise/bin/lwsm', 'status', 'vmdns']
2019-05-07T07:31:50.522Z Error: LW_ERROR_NO_SUCH_SERVICE (41204) No service with the specified name exists
2019-05-07T07:31:50.523Z VMAFD Boot failed

In /var/log/firstboot/vmafd-firstboot.py_stderr.log, you see entries similar to:

2019-05-07T07:31:50.524Z Exception: Traceback (most recent call last):
File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 177, in main
    controller.firstboot()
File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 51, in firstboot
    self.start()
File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 67, in start
    service.start()
File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmdnsInstall.py", line 50, in start
    self.utils.start_service(self.__VMDNS_SERVICE_NAME, self.__VMDNS_COMPONENT_NAME)
File "/usr/lib/vmware-vmafd/firstboot/identityinstall/LinuxUtils.py", line 131, in start_service
    raise ServiceStartException(serviceName)
cis.exceptions.ServiceStartException: {
    "componentKey": null,
    "problemId": null,
    "detail": [
        {
            "localized": "An error occurred while starting service 'VMwareDNSService'",
            "args": [
                "VMwareDNSService"
            ],
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "resolution": null
}

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

There are two DNS RPMs: vmware-dns-server and vmware-dns-client. The client RPM is installed on vCenter with external PSC node, while the server RPM is installed on PSC nodes. When the RPMs are installed, several configuration parameters are added to the "registry”. The client and server RPMs each register a different set of config parameters, however both sets are stored in the same location in the registry, so when the server RPM is installed, it should overwrite the client parameters with the server set.

This issue occurs when the vmware-dns-server RPM is correctly installed on the machine, but the registry values still correspond to the client settings, not the server settings. This causes the VMwareDNSService service fail to start, since the service can't find the server settings that it needs in the registry, causing the vmafd firstboot to fail.

Resolution

This issue is resolved in vCenter Server 6.7 U3a, available at VMware Downloads.

Workaround:
Note: Take snapshot of all the vCenter Servers and PSCs (which are part of ELM) in a Powered off state before proceeding with below steps:

To workaround this issue, proceed with below steps on the vCenter Server Appliance where Convergence failed :

Find the vmware-dns-server rpm:
- find / -name vmware-dns-server*.rpm
Remove the vmware-dns-server rpm
- rpm -ev --nodeps vmware-dns-RPMName-FromStep1 (without .rpm extension)

Example - rpm -ev --nodeps vmware-dns-server-1.0.0-12304056.x86_64

Install the vmware-dns-server rpm
- rpm -ivh --nodeps /root/velma/rpm/<vmware-dns-name-FromStep1.rpm>

Example - rpm -ivh --nodeps /root/velma/rpm/vmware-dns-server-1.0.0-12304056.x86_64.rpm

Stop VMAFD Service
- service-control --stop vmafdd
Recreate VECS DB from convergence process backup
- mv /storage/db/vmware-vmafd/afd.db /storage/db/vmware-vmafd/afd.db_backup
- Copy AFD DB from backup
  - ls /root/velma/rollback/ -l (list the files and identify the full name of afd.db backup with the converge date)
  - Copy the backup DB to /storage/db/vmware-vmafd folder with file name 'afd.db' (Example - cp /root/velma/rollback/2019-08-09_05-58-37afd.db /storage/db/vmware-vmafd/afd.db)
Start VMAFD Service
- service-control --start vmafdd
Modify file /etc/vmware/install-defaults/deployment.node.type from embedded to management
- vi /etc/vmware/install-defaults/deployment.node.type
- Modify the string embedded to management
- Save and Exit the VI editor (Press Esc Key followed by :wq! and Enter)
Modify file /etc/issue from embedded to Exernal
- vi /etc/issue
- Modify the string embedded to External
- Save and Exit the VI editor (Press Esc Key followed by :wq! and Enter)
Modify file /etc/vmware/deployment.node.type from embedded to management
- vi /etc/vmware/deployment.node.type
- Modify the string embedded to management
- Save and Exit the VI editor (Press Esc Key followed by :wq! and Enter)
Modify file /etc/applmgmt/appliance/appliance.conf from embedded to management
- vi /etc/applmgmt/appliance/appliance.conf
- Modify the string embedded to External
- Save and Exit the VI editor (Press Esc Key followed by :wq! and Enter)
Modify DomainState in Likewise Registry to 0x00000002
- /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmafd\Parameters]' DomainState 2
- Verify the modified value using list_values command - /opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmafd\Parameters]'
Recreate the Deleted Machine Account by following the Steps in KB Recreate vCenter Server Machine Account in Platform Services Controller after Failed Convergence
Rename the Convergence failed backup folder /root/velma
- mv /root/velma /root/velma_$(date +%d-%m-%Y_%H_%M_%S)
Rename failed Convergence logs under /var/log/vmware/converge/
- mv /var/log/vmware/converge/converge.log /var/log/vmware/converge/converge_$(date +%d-%m-%Y_%H_%M_%S).log
- mv /var/log/vmware/converge/converge_status.json /var/log/vmware/converge/converge_status_$(date +%d-%m-%Y_%H_%M_%S).json
- mv /var/log/vmware/converge/converge_rest.log /var/log/vmware/converge/converge_rest_$(date +%d-%m-%Y_%H_%M_%S).log
Start all the services
- service-control --start --all
Retry the Convergence

Additional Information

Deploying or Installing an additional Platform Service Controller 6.0 Update 1b fails during vmafd firstboot
Recreate vCenter Server Machine Account in Platform Services Controller after Failed Convergence