Unable to enable vSAN file service after vSAN cluster restart

Products

VMware vSAN

Issue/Introduction

Symptoms:

vCenter has no internet access.
vSAN cluster was shut down for planned maintenance.
vSAN File Services shows as disabled.
When going through the vSAN Files Service wizard you can't progress beyond the OVF file upload/verification.
During VCF upgrade, disabled vSAN File service and when trying to enable the service with "File service agent getting an error as: "No internet connection detected"and option disabled.

Validation:

From VC, /var/log/vmware/vSphere-client/logs, able to see timeout error.

[2022-10-13T18:40:11.539Z] [ERROR] http-nio-5090-exec-6 com.vmware.vsan.client.services.ProxygenController (r640f-fdd011d9bb20) Service method encountered an error caused by: `Failed to extract the requested data. Check vSphere Client logs for details.` com.vmware.vim.vmomi.client.exception.ConnectionException: https://vcsa1.lab.com:443/vsanHealth invocation failed with "java.net.SocketTimeoutException: Read timed out"
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:356)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:31)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)

From VC, var/log/vmware/vsan-health/vmware-vsan-health-service.log can see "failed to find OVF download URL"

2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf1c2] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061bf1c2] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T13:33:38.971Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Got minVersion 7.0.3 and maxVersion for 2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf1c2] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061bf1c2] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T13:33:38.971Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Got minVersion 7.0.3 and maxVersion for OVF version 7.0.3.1000-18502520
2022-10-18T13:33:38.972Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Found compatible OVF versions of {'7.0.3.1000-18502520': '7.0.3.1000-18502520'}
2022-10-18T13:33:39.315Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061bf200] Getting a file service OVF download URL...
2022-10-18T13:33:39.704Z INFO vsan-mgmt[08559] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf200] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:39.705Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061bf200] The compatible OVF version is 7.0.3
2022-10-18T13:33:39.705Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061bf200] Tring to find OVF download URL from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T13:36:01.732Z ERROR vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061bf200] Failed to find OVF download URL with version 7.0.3 from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T15:07:43.007Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061c028e] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T15:07:43.008Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T15:07:43.008Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061c028e] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T15:07:43.012Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Got minVersion 7.0.3 and maxVersion for OVF version 7.0.3.1000-18502520
2022-10-18T15:07:43.012Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Found compatible OVF versions of {'7.0.3.1000-18502520': '7.0.3.1000-18502520'}
2022-10-18T15:07:43.373Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061c02c9] Getting a file service OVF download URL...
2022-10-18T15:07:43.780Z INFO vsan-mgmt[56547] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061c02c9] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T15:07:43.781Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061c02c9] The compatible OVF version is 7.0.3
2022-10-18T15:07:43.781Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061c02c9] Tring to find OVF download URL from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T15:10:03.966Z ERROR vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061c02c9] Failed to find OVF download URL with version 7.0.3 from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

vCenter does not have internet access, and the OVF validation logic in the UI is dependent on online connectivity. Since the validation is not designed to function in offline, it fails when attempting to validate OVF files without internet access.

Resolution

1. Download all 6 vSAN file services files through the Broadcom Portal via KB: Download location of vSAN Witness Appliance and File Service Appliance

2. SSH into the vCenter appliance, navigate to directory: /storage/updatemgr/vsan/fileService

3. Make a new directory with name ovf-7.0.3.1000-18502520 . Example : mkdir ovf-7.0.3.1000-18502520

4. Use scp to copy all 6 OVF files to this new directory: /storage/updatemgr/vsan/fileService/ovf-7.0.3.1000-18502520/

5. Use command vmon-cli -r vsan-health to restart the vsan-health service. Also use command service-control restart eam to restart the eam service.

6. Login to the VC UI, go to vSAN cluster-> Configure and the enable vSAN file service.