How to remediate vSAN File Service VMs (FSVM) deployment issues after a major version upgrade of vCenter

Products

VMware vSAN

Issue/Introduction

After a major version upgrade of vCenter, File Service VMs may fail to be redeployed in the workflow of File Service Remediation.

In vCenter vsan-health-service.log you see the below messaging:
2023-09-27T13:27:13.213Z ERROR vsan-mgmt[47542] [VsanClusterFileServiceSystemImpl::_RemediateClusterFileServiceTask opID=10423f2b-W4041] Exception happened in deploying OVF in cluster 'vim.ClusterComputeResource:domain-c##'
Traceback (most recent call last):
File "bora/vsan/fileservice/vpxd/VsanClusterFileServiceSystemImpl.py", line 2551, in _RemediateClusterFileServiceTask
File "bora/vsan/fileservice/vpxd/VsanEamUtil.py", line 143, in DeployOvfInCluster
File "bora/vsan/fileservice/vpxd/VsanEamUtil.py", line 113, in SetupEamAgentDatastoreAndNetwork
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/pyEam/VmomiSupport.py", line 598, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/pyEam/VmomiSupport.py", line 388, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/pyEam/SoapAdapter.py", line 1575, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyEam.VmomiSupport.eam.fault.InvalidAgentConfiguration: (eam.fault.InvalidAgentConfiguration) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
invalidAgentConfiguration = (eam.Agent.ConfigInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
productLineId = <unset>,
hostVersion = '6+',
ovfPackageUrl = 'https://example.domain:443/vsanHealth/fileService/ovf/7.0.3.1000-18502520/VMware-vSAN-File-Services-Appliance-7.0.3.1000-18502520_OVF10.ovf',
ovfEnvironment = (eam.Agent.OvfEnvironmentInfo) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
ovfProperty = (eam.Agent.OvfEnvironmentInfo.OvfProperty) [
(eam.Agent.OvfEnvironmentInfo.OvfProperty) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = 'domain',
value = ''
},
(eam.Agent.OvfEnvironmentInfo.OvfProperty) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = 'searchpath',
value = ''
}
]
},
…
invalidField = 'ovfPackageUrl'
}

Environment

VMware vSAN (All Versions)

vSAN File Services (All Versions)

Cause

When manually upgrading vCenter to a major version (e.g. migrating vCenter from 8.0u2 to 9.0 builds), vcsa-iso needs to be downloaded to a windows jump worker , and then follow UI installer to upgrade vCenter to major release, FSVM ovf may be deleted in this way.

Resolution

Re-download the vSAN file service OVF files

To manually re-download vSAN file service OVF files, follow the steps below:

Download all missing OVF files. The file service OVF files required can be found in vsan-health-service.log error via ovfPackageUrl= as seen above. See KB Download location of vSAN Witness Appliance and File Service Appliance for downloading the files.
Create directory ovf-<ovf version> in this location /storage/updatemgr/vsan/fileService/ where <ovf version> is the version as specified via ovfPackageUrl= in the vsan-health-service.log error message above.

Example 1 (pre 7.0U2)
ovfPackageUrl = 'https://example.com:443/vsanHealth/fileService/ovf/7.0.0.1000/VMware-vSAN-File-Services-Appliance-7.0.0.1000-15817962_OVF10.ovf'
The <ovfversion> should be '7.0.0.1000' -> file directory /storage/updatemgr/vsan/fileService/ovf-7.0.0.1000/

Example 2 (7.0U2 and after)
ovfPackageUrl = 'https://example.domain:443/vsanHealth/fileService/ovf/7.0.3.1000-18502520/VMware-vSAN-File-Services-Appliance-7.0.3.1000-18502520_OVF10.ovf'
The <ovfversion> should be '7.0.3.1000-18502520' -> file directory /storage/updatemgr/vsan/fileService/ovf-7.0.3.1000-18502520/
Upload all downloaded OVF files in step 1 to vCenter under directory /storage/updatemgr/vsan/fileService/ovf-<ovf version>/ created via step 2 by using scp command.
Trigger RemediateFileService from UI.