vSAN File Services -- Infrastructure Health: 'File server not found' or 'vSAN File service not enabled'
search cancel

vSAN File Services -- Infrastructure Health: 'File server not found' or 'vSAN File service not enabled'

book

Article ID: 372631

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

vSAN Health Alarm: Infrastructure Health: 





The alert details might show one or more of the following:
  • File service VM not found on this host (might appear for more than one Host)
  • Sockrelay is not running

 

In addition, one or more of the following symptoms might be experienced:

  • The vSAN Health Alarm is not going away after hitting "Remediate" in the Alarm details
  • Prior observing the vSAN Health alert, there was a Maintenance activity on the reported Host(s)

  • vSphere Client: Remediating the vSAN file services fails with below error: 

Remediate vSAN file service: "Unable to enable the vSAN file service: FSVM does not exist from the beginning"

 

  • vSphere Client: Remediating the vSAN file services fails with below error:

Remediate vSAN file service: "Cannot complete the operation. See the event log for details. Unable to enable the vSAN file service."

 

  • Attempting to restart the fsvmsockrelay service, fails with the below error:

/etc/init.d/fsvmsockrelay restart
sockrelay is not running
No fsvm-sockrelay resource pool found
vSAN File Service Node cannot be found on this host

  • vSphere Client: "Install agent" tasks by vCenter EAM service fail with the error  "Unable to access agent OVF package file"

Environment

VMware vSAN 7.x
VMware vSAN 8.x
VMware vSAN 9.x

Cause

vCenter has missing OVF and related files for vSAN File Services VMs in the designated path, which can be determined by the following messages in the vCenter logs.
Issue is often observed in relation to completed vCenter upgrades (e.g. from 7.x to 8.x) where OVF files are not retained from older version
 

/var/log/vmware/eam/eam.log:

Example output

YYYY-MM-DDTHH:MM:SS.ZZ |  WARN | vlsi | Workflow.java | 156 | [OvfValidator->Validate:http://localhost:1080/external-tp/httpl/hostname.domain.com/443/e6714aaec7d3ffef1e34cd0c8e2621fe67410cff/vsanHealth/fileService/ovf/7.0.3.1000-20036589/V
Mware-vSAN-File-Services-Appliance-7.0.3.1000-20036589_OVF10.ovf:d6568d69a9a44e0b] NEXT WORK ITEM : Failed to instantiate
com.vmware.eam.exception.CannotAccessOVF: Cannot access OVF at http://localhost:1080/external-tp/http1/hostname.domain.com/443/e6714aaec7d3ffef1e34cd0c8e2621fe67410cff/vsanHealth/fileService/ovf/7.0.3.1000-20036589/VMware-vSAN-File-Services-Appliance-7.0.3.1000-20036589_OVF10.ovf

/var/log/vmware/vsan-health/vmware-vsan-health-service.log

Example output:

YYYY-MM-DDTHH:MM:SS.ZZ ERROR vsan-mgmt[12761] [VsanHttpProvider::doGet opID=noOpId] Looking for non-existing path /storage/vsan-health/../updatemgr/vsan/fileService/ovf-7.0.3.1000-20036589/VMware-vSAN-File-Services-Appliance-7.0.3.1000-20036589_OVF10.ovf, return 404

YYYY-MM-DDTHH:MM:SS.ZZ INFO vsan-mgmt[12761] [VsanMgmtServer::log_message opID=noOpId] ('127.0.0.1', 52490) - - "GET /vsanHealth/fileService/ovf/7.0.3.1000-20036589/VMware-vSAN-File-Services-Appliance-7.0.3.1000-20036589_OVF10.ovf HTTP/1.1" 404 -

YYYY-MM-DDTHH:MM:SS.ZZ ERROR vsan-mgmt[09194] [VsanClusterFileServiceSystemImpl::_RemediateClusterFileServiceTask opID=77504a72-W3314] Exception happened in deploying OVF in cluster 'vim.ClusterComputeResource:domain-##'

 

Resolution

1.) Download the FSVM version OVF and the other files for vSAN File service again from the Broadcom portal.
(My Downloads > Enter 'vSphere' in 'Search Product Name' field > Click on vSAN > Click on the the appropriate vSphere/ESXi version > Go to 'Drivers & Tools' tab > Select the appropriate vSAN File Service VM OVF is required )

There should be a total of 6 files:

VMware-vSAN-File-Services-Appliance-#.#.#.#-#_OVF10.mf
VMware-vSAN-File-Services-Appliance-#.#.#.#-#-#_OVF10.cert
VMware-vSAN-File-Services-Appliance-#.#.#.#-#-#-system.vmdk
VMware-vSAN-File-Services-Appliance-#.#.#.#-#-cloud-components.vmdk
VMware-vSAN-File-Services-Appliance-#.#.#.#-#-log.vmdk
VMware-vSAN-File-Services-Appliance-#.#.#.#-#_OVF10.ovf


2.) Create the following path: 

/storage/updatemgr/vsan/fileService/ovf-x.y.z.aaaa-bbbbbbb

Example: /storage/updatemgr/vsan/fileService/ovf-7.0.3.1000-20036589

(where ovf-x.y.z.aaaa-bbbbbbb is the version found from /var/log/vmware/vsan-health/vmware-vsan-health-service.log
Alternatively, you can find the version in vSphere Client from vSAN Cluster > Configure > Services > File Service )

 
 
3.) Transfer the downloaded files to the newly created path using WinSCP or any file transfer utility

 
4.) Return to vCenter SSH, navigate to path /storage/updatemgr/vsan and modify the User and Group owner for the directory 'fileService' and directories under it:
Command to run :

chown -R vsan-health:users fileService

 
5.) Run following commands to change permissions on these directories

chmod 755 /storage/updatemgr/vsan/fileService
chmod 644 /storage/updatemgr/vsan/fileService/ovf-x.y.z.aaaa-bbbbbbb
 
6.) Proceed to remediate the file service from Skyline Health under 'Infrastructure Health' option

This action will enable the EAM agent to deploy the FSVM.
(vSAN Cluster > Monitor > vSAN Skyline Health > Infrastructure Health > Troubleshoot)

7.) Re-run 'Retest' vSAN Skyline Health from vSphere Client
(vSAN Cluster > Monitor > vSAN Skyline Health)

 

 

 

 

 

 

 

 

 

 

 

Additional Information

If assistance is required, open a Ticket with VMware by Broadcom Support.

Powered by Wolken Software