vSAN File shares inaccessible after hosts moved between vSAN clusters with vSAN file services enabled.
Article ID: 389332
Updated On:
Products
Issue/Introduction
Symptoms:
vSAN file shares may become inaccessible after upgrading and rebooting all the hosts in the vSAN cluster with file service enabled.
vSAN file services health may report incorrect configuration information.
The file share capacity on vCenter under File Shares tab show 0.00 bytes in size.
FSVMs would be deploying/working as expected, but the containers may continue to restart inside the FSVMs.
Environment
VMware vSAN 7.x
VMware vSAN 8.x
Cause
The issue would occur If the vSAN host/s moved between the vSAN clusters with vSAN file services enabled on source (from which vSAN cluster the host was removed) and destination (to which vSAN cluster host is added) cluster. The ESXi host was removed from the vSAN cluster, but the disk groups were not deleted before adding the host to a new vSAN cluster where the vSAN file service is already enabled.
This would create two root filesystems for vsanfs which causes the file share and infra health would impact. The root filesystem can be identified under vdfs datastore path on ESXi host.
/vmfs/volumes/vdfs:0000000000000000-0000000000000000/
Example: We see the below two paths under vdfs datastore.
When check the logs on ESXI host /var/run/log/vsanfs-mgmt.log report DNS lookup issues.
2025-02-08T02:35:41.329Z In(14) vsanfs.mgmt[4181760]: [4181787] [VDFSEndpointContainerDocker::_ExecRun] Running cmd nslookup -ret=1 -timeout=5 fs01.example.com 172.xx.141.227 on container None(fs01), with timeout: 6
2025-02-08T02:35:41.332Z Er(11) vsanfs.mgmt[4181760]: [4181787] [VDFSEndpointContainerDocker::_ExecRun] Failed to run cmd /usr/bin/timeout --signal=SIGKILL 6 nslookup -ret=1 -timeout=5 fs01.example.com 172.xx.141.227 on fs01(None)
2025-02-08T02:35:41.332Z Er(11) vsanfs.mgmt[4181760]: [4181787] [VDFSEndpointMonitorContainer::_CheckContainerHealthItem] CONT: not found on checking DnsLookupTest for container 172.XX.155.162
When reviewed the configuration, we see different Network configuration on config paths with different root filesystems. In this example you will see 172.XX.155.xx and 172.XX.187.xx IP configured in two different configuration files. This can be reviewed on ESXi host under the vdfs datastore path.
Cause can be validated using the vdfs configuration files under root filesystem.
- .Primary set of information on VDFS Datastore:
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/fb227e63-fcf3-f355-1d69-##########/volumes/26fe5d42-53e3-b75b-1029-##########/default/config.json] cat 2/data
{"version": "2.1.0", "domainInfos": {"6b95d0a4-65dd-4005-95c3-##########": {"fsDomainName": "vsanfs", "dnsAddresses": ["172.xx.189.227", "172.xx.173.227"], "dnsSuffixes": ["###.net"], "primaryIP": "172.xx.187.162", "directoryServerConfig": {}, "fileServerIpConfig": [{"dhcp": false, "ipAddress": "172.xx.187.162", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs01.example.com", "primary": true, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.163", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs02.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.164", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs03.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.165", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs04.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.166", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs05.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.167", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs06.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.168", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs07.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.169", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs08.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}, {"dhcp": false, "ipAddress": "172.xx.187.170", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs09.example.com", "primary": false, "gateway": "172.xx.187.161", "affinityLocation": "None"}]}}}
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/fb227e63-fcf3-f355-1d69-##########/volumes/26fe5d42-53e3-b75b-1029-##########/default/config.json]
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/fb227e63-fcf3-f355-1d69-##########/volumes/26fe5d42-53e3-b75b-1029-##########/default/config_6b95d0a4-65dd-4005-95c3-##########.json] cat 248/data
{"host-fscontainer-map": {"630ce1d9-000f-c0f0-f159-##########": ["172.xx.187.162"], "630cf604-fb59-6304-6b81-##########": ["172.xx.187.168"], "630ce1a2-8fe5-0a4c-677d-##########": ["172.xx.187.164"], "66e173d5-8167-fd16-e316-##########": ["172.xx.187.165"], "66e17150-797b-f486-73aa-##########": ["172.xx.187.167"], "6298bd36-55c1-b5e4-b76d-##########": ["172.xx.187.166"], "62961d6a-4932-fabc-9388-##########": ["172.xx.187.163"]}, "fscontainer-properties": {"172.xx.187.162": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.163": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.164": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.165": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.166": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.167": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.168": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.187.169": {"fscontainerState": "REMOVING", "fscontainerWaiter": "62961d6a-4932-fabc-9388-##########", "affinityLocation": "None"}}, "version": "2.1.0", "shard-ip-map": {"file:c7d458ce-642f-4509-a28a-##########": {"ed237e63-beed-7880-de6d-##########": "172.xx.187.162"}, "file:0ac55b20-055d-4add-a8b2-##########": {"11783467-9267-af7c-2c79-##########": "172.xx.187.163"}}, "share-properties": {"file:c7d458ce-642f-4509-a28a-##########": {"state": "READY", "type": 0, "quota": "0", "storagePolicy": "{'share': '((\"spbmProfileId\" \\'a24bc629-9fa0-47d5-84d1-##########\\' )(\"spbmProfileGenerationNumber\" l+1 )(\"spbmProfileName\" \\'Storage Policy FTT-1 RAID-1 STRIPE-2\\' )(\"iopsLimit\" i0 ) (\"forceProvisioning\" i0 ) (\"replicaPreference\" \"Performance\" ) (\"hostFailuresToTolerate\" i1 ) (\"checksumDisabled\" i0 ) (\"proportionalCapacity\" i0 ) (\"stripeWidth\" i2 ) (\"cacheReservation\" i0 ) )'}", "name": "share01", "softQuota": "0", "protocols": ["NFSv4"], "secType": "SYS", "smbOptions": null, "affinityLocation": "None", "hasLabel": false, "permissions": [{"ips": "*", "permissions": "READ_WRITE", "allowRoot": true}], "shareWaiter": "6298bd36-55c1-b5e4-b76d-##########"}, "file:0ac55b20-055d-4add-a8b2-##########": {"state": "READY", "type": 0, "quota": "1T", "storagePolicy": "{'share': '((\"spbmProfileId\" \\'a24bc629-9fa0-47d5-84d1-##########\\' )(\"spbmProfileGenerationNumber\" l+1 )(\"spbmProfileName\" \\'Storage Policy FTT-1 RAID-1 STRIPE-2\\' )(\"cacheReservation\" i0 ) (\"forceProvisioning\" i0 ) (\"iopsLimit\" i0 ) (\"proportionalCapacity\" i0 ) (\"replicaPreference\" \"Performance\" ) (\"stripeWidth\" i2 ) (\"hostFailuresToTolerate\" i1 ) (\"checksumDisabled\" i0 ) )'}", "name": "share02", "softQuota": "1T", "protocols": ["NFSv3", "NFSv4"], "secType": "SYS", "smbOptions": null, "affinityLocation": "None", "hasLabel": false, "permissions": [{"ips": "*", "permissions": "READ_WRITE", "allowRoot": true}], "shareWaiter": "62963c4c-ec4d-a62c-0d0d-##########"}}, "referral-shares": {"172.xx.187.162": [["ed237e63-beed-7880-de6d-##########", "file:c7d458ce-642f-4509-a28a-##########", "share01"]], "172.xx.187.163": [["11783467-9267-af7c-2c79-##########", "file:0ac55b20-055d-4add-a8b2-##########", "share02"]]}, "next-export-id": 104}
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/fb227e63-fcf3-f355-1d69-##########/volumes/26fe5d42-53e3-b75b-1029-##########/default/config_6b95d0a4-65dd-4005-95c3-##########.json]
- Secondary set of information on VDFS Datastore:
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/d9f1f363-24bd-ee1d-7bd2-##########/volumes/44972e5f-4b8f-9977-83f2-##########/default/config.json]
cat 8/data
{"version": "2.3.0", "domainInfos": {"83025203-c20c-4a17-a952-##########": {"fsDomainName": "new-vsanfs", "dnsAddresses": ["172.xx.141.227", "172.xx.141.228"], "dnsSuffixes": ["###.net"], "primaryIP": "172.xx.155.162", "directoryServerConfig": {}, "fileServerIpConfig": [{"dhcp": false, "ipAddress": "172.xx.155.162", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs01.example.net", "primary": true, "gateway": "172.xx.155.161", "affinityLocation": null}, {"dhcp": false, "ipAddress": "172.xx.155.163", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs02.example.net", "primary": false, "gateway": "172.xx.155.161", "affinityLocation": null}, {"dhcp": false, "ipAddress": "172.xx.155.164", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs03.example.net", "primary": false, "gateway": "172.xx.155.161", "affinityLocation": null}, {"dhcp": false, "ipAddress": "172.xx.155.165", "subnetMask": "255.255.255.224", "ipV6Config": {}, "fqdn": "fs04.example.net", "primary": false, "gateway": "172.xx.155.161", "affinityLocation": null}]}}, "updated-by": "629dcc4f-d73a-0352-0dd5-########"}
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/d9f1f363-24bd-ee1d-7bd2-##########/volumes/44972e5f-4b8f-9977-83f2-##########/default/config.json]
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/d9f1f363-24bd-ee1d-7bd2-##########/volumes/44972e5f-4b8f-9977-83f2-##########/default/config_83025203-c20c-4a17-a952-##########.json]
cat 12311/data
{"host-fscontainer-map": {"66e173d5-8167-fd16-e316-##########": ["172.xx.155.164"], "6298bd36-55c1-b5e4-b76d-##########": ["172.xx.155.163"], "630cf604-fb59-6304-6b81-##########": ["172.xx.155.165"], "62963c4c-ec4d-a62c-0d0d-##########": ["172.xx.155.162"]}, "fscontainer-properties": {"172.xx.155.162": {"fscontainerState": "READY", "affinityLocation": "None", "fscontainerWaiter": "62961d6a-4932-fabc-9388-##########", "failoverState": "FAILING_OVER", "failoverWaiter": "62963c4c-ec4d-a62c-0d0d-##########"}, "172.xx.155.163": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.155.164": {"fscontainerState": "READY", "affinityLocation": "None"}, "172.xx.155.165": {"fscontainerState": "READY", "affinityLocation": "None"}}, "version": "2.3.0", "shard-ip-map": {"file:61f5f45b-144a-46d4-9517-##########": {"83f5f363-8497-c4dd-a5e3-##########": "172.xx.155.162"}}, "share-properties": {"file:61f5f45b-144a-46d4-9517-##########": {"state": "READY", "type": 0, "quota": "0", "name": "share01", "softQuota": "0", "protocols": ["NFSv4"], "secType": "SYS", "smbOptions": null, "affinityLocation": "None", "hasLabel": false, "permissions": [{"ips": "*", "permissions": "READ_WRITE", "allowRoot": true}], "shareWaiter": "630ce1a2-8fe5-0a4c-677d-##########", "storagePolicy": "2a72a907-d57c-41c2-980a-##########"}}, "storage-policies": {"2a72a907-d57c-41c2-980a-##########": "{'share': '((\"cacheReservation\" i0)(\"checksumDisabled\" i0)(\"forceProvisioning\" i0)(\"hostFailuresToTolerate\" i1)(\"iopsLimit\" i0)(\"proportionalCapacity\" i0)(\"replicaPreference\" \"Performance\")(\"spbmProfileGenerationNumber\" l+1)(\"spbmProfileId\" \\'a24bc629-9fa0-47d5-84d1-##########\\')(\"spbmProfileName\" \\'Storage Policy FTT-1 RAID-1 STRIPE-2\\')(\"stripeWidth\" i2))'}"}, "updated-by": "62963c4c-ec4d-a62c-0d0d-##########"}
[esxsu@ESXi:/vmfs/volumes/vdfs:0000000000000000-0000000000000000/d9f1f363-24bd-ee1d-7bd2-##########/volumes/44972e5f-4b8f-9977-83f2-##########/default/config_83025203-c20c-4a17-a952-##########.json]
Example below:
- The same file share "
share01" is associated with two different root filesystems.
rootfs volume: fb227e63-fcf3-f355-1d69-########## sharename: share01 contaienrIP: 172.xx.187.162 containerFQDN: pv###fs01.example.comrootfs volume: d9f1f363-24bd-ee1d-7bd2-########## sharename: share01 containerIP: 172.xx.155.162 containerFQDN: dv###fs01.example.net
When we have two root filesystems under VDFS, filesystem would fail to come up the file shares due to configuration mismatch.
Resolution
It is always recommended to follow the instructions in the article Permanently Decommissioning a node from a vSAN Cluster to remove and add hosts between clusters. This avoid any additional configuration/data coming on to a destination cluster where the host is added.
It is required to fix the stale root fs which was carried along with the hosts disk group when the ESXi host moved from vSAN FS enabled cluster to another vSAN FS enabled cluster.
Please contact Broadcom support for further assistance on this if the symptoms match.
Additional Information
Feedback
