Error: "File server is (re)starting" seen after configuring vSAN FS

Products

VMware vSAN

Issue/Introduction

vSAN FS is configured on a stretched cluster
This is a new configuration and has never worked
vSANFS fails to configure on secondary site of vSAN cluster only
NFS is working on the primary/preferred site.
From /var/run/log/vsanfs.mgmt.log,entries of the following type are seen:

Wa(12) vsanfs.mgmt[2099685]: [EndpointController-D-1] [VDFSEndpointController::WaitObjects] Timeout to wait objects: ['fscontainer/failover/<fs-vm-ip>']
Er(11) vsanfs.mgmt[2099685]: [EndpointController-D-1] [VDFSEndpointController::WaitObjects] Failed to wait objects ['fscontainer/failover/<fs-vm-ip>']
Er(11)[+] vsanfs.mgmt[2099685]: Traceback (most recent call last):
Er(11)[+] vsanfs.mgmt[2099685]: File "/usr/lib/vmware/vsan/perfsvc/VDFSEndpointController.py", line 3002, in WaitObjects
Er(11)[+] vsanfs.mgmt[2099685]: PyCppVmomi.vim.fault.VimFault: (vim.fault.VimFault) {
Er(11)[+] vsanfs.mgmt[2099685]: faultMessage = (vmodl.LocalizableMessage) [
Er(11)[+] vsanfs.mgmt[2099685]: (vmodl.LocalizableMessage) {
Er(11)[+] vsanfs.mgmt[2099685]: key = 'com.vmware.vsan.fileservice.fault.operation.timeout',
Er(11)[+] vsanfs.mgmt[2099685]: arg = (vmodl.KeyAnyValue) [
Er(11)[+] vsanfs.mgmt[2099685]: (vmodl.KeyAnyValue) {
Er(11)[+] vsanfs.mgmt[2099685]: key = 'reason',
Er(11)[+] vsanfs.mgmt[2099685]: value = "Timeout to wait objects ['fscontainer/failover/<fs-vm-ip>']"
Er(11)[+] vsanfs.mgmt[2099685]: }
Er(11)[+] vsanfs.mgmt[2099685]: ],
Er(11)[+] vsanfs.mgmt[2099685]: message = "Timeout to wait objects ['fscontainer/failover/<fs-vm-ip>']"
Er(11)[+] vsanfs.mgmt[2099685]: }
Er(11)[+] vsanfs.mgmt[2099685]: ]
Er(11)[+] vsanfs.mgmt[2099685]: }
In(14) vsanfs.mgmt[2099685]: [EndpointController-D-1] [VDFSEndpointContainerDocker::Create] CONT: starting FSContainer <fs-vm-name>
Er(11) vsanfs.mgmt[2099685]: [EndpointController-D-1] [VDFSEndpointContainerDocker::Create] CONT: Warning: failed to create container
From the Fileserver VM logs on the secondary site,/scratch/log/vdfs_support/containers/fs_vm_logs/fsvm_logs/containers/<FS VM name>/ganesha/ganesha.log, the NFS service is seen as initialized and immediately stopped.
ganesha.nfsd-104[none] [main] 941 :nfs_start :NFS STARTUP :-------------------------------------------------
ganesha.nfsd-104[none] [main] 942 :nfs_start :NFS STARTUP : NFS SERVER INITIALIZED
ganesha.nfsd-104[none] [main] 944 :nfs_start :NFS STARTUP :-------------------------------------------------
ganesha.nfsd-104[none] [Admin] 596 :do_shutdown :MAIN :NFS EXIT: stopping NFS service
From the vCenter server(VC) logs /var/log/vmware/eam/eam.log,the service reports the FS VM deployment issue identified and will repeat for each host:

INFO | vim-inv-update | IssueHandlerBase.java | 116 | Updating issues:
New issues:
[
eam.issue.VmNotDeployed {
time = <Timestamp>,
description = <unset>,
key = 48##,
agency = 'Agency:fd######-####-####-####-############:null',
agencyName = 'vsan-file-services',
solutionId = 'com.vmware.vsan.health',
solutionName = 'com.vmware.vsan.health',
agent = 'Agent:12######-####-####-####-############:null',
agentName = '12######-####-####-####-############',
host = 'HostSystem:<host-moref>:########-####-####-####-############',
hostName = '<ESXi host name>',
}
]
Removed issues: []
FSContainer's are in restarting loop on all the hosts in the secondary site, as seen from /var/log/vmware/vsan-health/vsan-health.log logs:

Test fileserver health : red
FileServerConnectivity: Domain IpAddress Host Network DnsLookup Description
(Vfs-Mgmt, <Primary site host IP>, <Primary site Host-moref>, Green, Green, FileServerConnectivityIsInGoodState.), (Vfs-Mgmt, <Primary site host IP>, <Primary site Host-moref>, Green, Green, FileServerConnectivityIsInGoodState.),
(Vfs-Mgmt, <Primary site host IP>, <Primary site Host-moref>, Green, Green, FileServerConnectivityIsInGoodState.), (Vfs-Mgmt, <Secondary site host IP>, <Secondary site Host-moref>, Red, Red, FileServerIs(Re)Starting.),
(Vfs-Mgmt, <Secondary site host IP>, <Secondary site Host-moref>, Red, Red, FileServerIs(Re)Starting.), (Vfs-Mgmt, <Secondary site host IP>, <Secondary site Host-moref>, Red, Red, FileServerIs(Re)Starting.),
The container VM is seen as created on one host and simultaneously another thread about deleting the FS VM is seen on another host just after 2 seconds:
<date>T12:10:59.513Z In(14) vsanfs.mgmt[########]: [EndpointController-D-1] [VDFSEndpointContainerDocker::Create] CONT: creating FSContainer <fs-vm-name>
<date>T12:10:59.537Z In(14) vsanfs.mgmt[########]: [EndpointController-D-1] [VDFSEndpointController::_WaitForDestroyFSContainer] CONT: waiting for container <fs-vm-ip> being destroyed from remote host
<date>T12:11:06.932Z In(14) vsanfs.mgmt[########]: [2099848] [VDFSEndpointMonitor::_GetUnexpectedContainers] FSContainer <fs-vm-name>(<fs-vm-ip>) is unexpected on the host
<date>T12:11:06.932Z In(14) vsanfs.mgmt[########]: [2099848] [VDFSEndpointContainerDocker::Destroy] CONT: Destroying container None(<fs-vm-name>)
<date>T12:11:06.943Z In(14) vsanfs.mgmt[########]: [2099848] [VDFSEndpointContainerDocker::Destroy] FS Container <fs-vm-name> is destroyed
<date>T12:11:44.558Z In(14) vsanfs.mgmt[########]: [EndpointController-D-1] [VDFSEndpointContainerDocker::Create] CONT: starting FSContainer <fs-vm-name>
<date>T12:11:44.561Z Er(11) vsanfs.mgmt[########]: [EndpointController-D-1] [VDFSEndpointContainerDocker::Create] CONT: Warning: failed to create container

Environment

VMware vCenter Server 8.x

VMware vSAN 8.x

Cause

The issue is due to FS VM container monitor regression. The container creation in new host needs to wait for the container deletion on old host. However, the container failover happened in very few seconds, the old host had no chance to create the container,delete the container, and notify the deletion result to new host. Timeout when waiting for notification is the side effect of continuous failover in seconds.

Resolution

To resolve the issue, restart the vsanmgmt service on all the hosts in the secondary site to clear the dirty state of the container monitor.

SSH to each of the host in the secondary site.
Run the following command :

/etc/init.d/vsanmgmtd restart
Verify the vSAN Skyline Health to check the status is healthy.