Unable to enable vSAN file service after vSAN cluster restart
Article ID: 314300
Updated On:
Products
VMware vSAN
Issue/Introduction
Impact/Risks:
vSAN File Services can't be enabled and can't access existing shares
vSAN File Services can't be enabled and can't access existing shares
Symptoms:
The environment is running version 7.0U3
vCenter has no internet access
vSAN cluster was shut down for planned maintenance
vSAN File Services shows as disabled
When going through the vSAN Files Service wizard you can't progress beyond the OVF file upload/verifaction
In the vsphere_client_virgo.log you see:
[2022-10-13T18:40:11.539Z] [ERROR] http-nio-5090-exec-6 com.vmware.vsan.client.services.ProxygenController (r640f-fdd011d9bb20) Service method encountered an error caused by: `Failed to extract the requested data. Check vSphere Client logs for details.` com.vmware.vim.vmomi.client.exception.ConnectionException: https://vcsa1.lab.com:443/vsanHealth invocation failed with "java.net.SocketTimeoutException: Read timed out"
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:356)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57)
at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:31)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
In the vmware-vsan-health-service.log you see:
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf1c2] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061bf1c2] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T13:33:38.971Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Got minVersion 7.0.3 and maxVersion for 2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf1c2] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061bf1c2] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T13:33:38.971Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Got minVersion 7.0.3 and maxVersion for OVF version 7.0.3.1000-18502520
2022-10-18T13:33:38.972Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Found compatible OVF versions of {'7.0.3.1000-18502520': '7.0.3.1000-18502520'}
2022-10-18T13:33:39.315Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061bf200] Getting a file service OVF download URL...
2022-10-18T13:33:39.704Z INFO vsan-mgmt[08559] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf200] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:39.705Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061bf200] The compatible OVF version is 7.0.3
2022-10-18T13:33:39.705Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061bf200] Tring to find OVF download URL from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T13:36:01.732Z ERROR vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061bf200] Failed to find OVF download URL with version 7.0.3 from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T15:07:43.007Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061c028e] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T15:07:43.008Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T15:07:43.008Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061c028e] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T15:07:43.012Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Got minVersion 7.0.3 and maxVersion for OVF version 7.0.3.1000-18502520
2022-10-18T15:07:43.012Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Found compatible OVF versions of {'7.0.3.1000-18502520': '7.0.3.1000-18502520'}
2022-10-18T15:07:43.373Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061c02c9] Getting a file service OVF download URL...
2022-10-18T15:07:43.780Z INFO vsan-mgmt[56547] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061c02c9] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T15:07:43.781Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061c02c9] The compatible OVF version is 7.0.3
2022-10-18T15:07:43.781Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061c02c9] Tring to find OVF download URL from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T15:10:03.966Z ERROR vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061c02c9] Failed to find OVF download URL with version 7.0.3 from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
The environment is running version 7.0U3
vCenter has no internet access
vSAN cluster was shut down for planned maintenance
vSAN File Services shows as disabled
When going through the vSAN Files Service wizard you can't progress beyond the OVF file upload/verifaction
In the vsphere_client_virgo.log you see:
[2022-10-13T18:40:11.539Z] [ERROR] http-nio-5090-exec-6 com.vmware.vsan.client.services.ProxygenController (r640f-fdd011d9bb20) Service method encountered an error caused by: `Failed to extract the requested data. Check vSphere Client logs for details.` com.vmware.vim.vmomi.client.exception.ConnectionException: https://vcsa1.lab.com:443/vsanHealth invocation failed with "java.net.SocketTimeoutException: Read timed out"
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:356)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57)
at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:31)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
In the vmware-vsan-health-service.log you see:
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf1c2] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061bf1c2] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T13:33:38.971Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Got minVersion 7.0.3 and maxVersion for 2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf1c2] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T13:33:38.967Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061bf1c2] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T13:33:38.971Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Got minVersion 7.0.3 and maxVersion for OVF version 7.0.3.1000-18502520
2022-10-18T13:33:38.972Z INFO vsan-mgmt[35142] [VsanFileServiceOVF::FindHighestOvfVersion opID=061bf1c2] Found compatible OVF versions of {'7.0.3.1000-18502520': '7.0.3.1000-18502520'}
2022-10-18T13:33:39.315Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061bf200] Getting a file service OVF download URL...
2022-10-18T13:33:39.704Z INFO vsan-mgmt[08559] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061bf200] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T13:33:39.705Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061bf200] The compatible OVF version is 7.0.3
2022-10-18T13:33:39.705Z INFO vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061bf200] Tring to find OVF download URL from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T13:36:01.732Z ERROR vsan-mgmt[08559] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061bf200] Failed to find OVF download URL with version 7.0.3 from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T15:07:43.007Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061c028e] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T15:07:43.008Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Checking OVF versions with compatible version 7.0.3 in preflight check ...
2022-10-18T15:07:43.008Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::ListFileServiceOvfVersions opID=061c028e] Available FSVM OVF versions are: ['7.0.3.1000-18502520']
2022-10-18T15:07:43.012Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Got minVersion 7.0.3 and maxVersion for OVF version 7.0.3.1000-18502520
2022-10-18T15:07:43.012Z INFO vsan-mgmt[10435] [VsanFileServiceOVF::FindHighestOvfVersion opID=061c028e] Found compatible OVF versions of {'7.0.3.1000-18502520': '7.0.3.1000-18502520'}
2022-10-18T15:07:43.373Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061c02c9] Getting a file service OVF download URL...
2022-10-18T15:07:43.780Z INFO vsan-mgmt[56547] [VsanFileServiceOVF::FindCompatibleVersionForCluster opID=061c02c9] Host version is 7.0.203, disk format is 15, vcVersion is 7.0.3.
2022-10-18T15:07:43.781Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::FindOvfDownloadUrl opID=061c02c9] The compatible OVF version is 7.0.3
2022-10-18T15:07:43.781Z INFO vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061c02c9] Tring to find OVF download URL from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
2022-10-18T15:10:03.966Z ERROR vsan-mgmt[56547] [VsanClusterFileServiceSystemImpl::_FindFsOvfUrlFromMap opID=061c02c9] Failed to find OVF download URL with version 7.0.3 from map: https://download3.vmware.com/software/VSANOVF/FsOvfMapping.json
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware vSAN 7.0.x
Cause
vCenter doesn't have internet access and the UI code logic for OVF validation errors out as it doesn't know how to handle the validation without internet access.
Resolution
Upgrade vCenter/ESXi to version 7.0U3l or higher
Workaround:
To workaround this issue ensure vCenter has internet access or if internet access is not an option follow the below steps:
1. SSH into the vCenter appliance, navigate to directory: /storage/updatemgr/vsan/fileService
2. make a new directory with name ovf-7.0.3.1000-18502520: mkdir ovf-7.0.3.1000-18502520
3. use scp to copy all 6 OVF files to this new directory: /storage/updatemgr/vsan/fileService/ovf-7.0.3.1000-18502520/
4. on the vCenter server, edit file /etc/vmware-vsan-health/config.conf, add the following lines at the bottom of the file
[FileService]
ovfurl = https://download3.vmware.com/software/VSAN-FILESERVICE-70-U3F/VMware-vSAN-File-Services-Appliance-7.0.3.1000-18502520_OVF10.ovf
5. execute command 'vmon-cli -r vsan-health' to restart the vsan-health service
6. relogin to the vCenter Web Client and open the enable vSAN file service wizard. If the OVF version you saved on the VC server is 18502520, the 'file service agent' step should not appear. If the version is not 18502520, the step appears, and there will be 3 options in the view, please choose 'Use current OVF'.
Workaround:
To workaround this issue ensure vCenter has internet access or if internet access is not an option follow the below steps:
1. SSH into the vCenter appliance, navigate to directory: /storage/updatemgr/vsan/fileService
2. make a new directory with name ovf-7.0.3.1000-18502520: mkdir ovf-7.0.3.1000-18502520
3. use scp to copy all 6 OVF files to this new directory: /storage/updatemgr/vsan/fileService/ovf-7.0.3.1000-18502520/
4. on the vCenter server, edit file /etc/vmware-vsan-health/config.conf, add the following lines at the bottom of the file
[FileService]
ovfurl = https://download3.vmware.com/software/VSAN-FILESERVICE-70-U3F/VMware-vSAN-File-Services-Appliance-7.0.3.1000-18502520_OVF10.ovf
5. execute command 'vmon-cli -r vsan-health' to restart the vsan-health service
6. relogin to the vCenter Web Client and open the enable vSAN file service wizard. If the OVF version you saved on the VC server is 18502520, the 'file service agent' step should not appear. If the version is not 18502520, the step appears, and there will be 3 options in the view, please choose 'Use current OVF'.
Additional Information
Feedback
Yes
No
Powered by
