Attempting to add a host to a Workload Domain is stuck as "In Progress" while attempting to enter / exit maintenance mode.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Add host workflow is in running state for a number of hours.
• Enter / Exit maintenance mode of a host is also still running when viewing the vCenter Web Client. 

Environment

VMware Cloud Foundation 4.0.x

Cause

The task to exit maintenance mode of a host gets stuck executing on vCenter and ends up taking up to 24 hours. This results in the SDDC Manager workflow taking multiple days before failing the workflow due to retry attempts.

Resolution

1. Open a SSH session to the SDDC Manager and switch to root:

su -

2.  Open the "application-prod.properties" file located in the /etc/vmware/vcf/domainmanager directory

vi /etc/vmware/vcf/domainmanager/application.properties

3.  Configure the "host.enterOrExitMaintenanceMode.timeoutInHours" property. Note: adjust the value to best suit the environment.

host.enterOrExitMaintenanceMode.timeoutInHours=20

4.  Check that there are no workflows running.
5.  Restart the domainmanager service:

systemctl restart doaminmanager

Workaround:
To workaround this issue, manually exit the hosts out of maintenance mode. This will fail the operation and allow the task to be retried.