vSAN File Service fails with Error " The File service is not available, because the file service domain is not successfully created. Click the edit button to create the file service domain again."
search cancel

vSAN File Service fails with Error " The File service is not available, because the file service domain is not successfully created. Click the edit button to create the file service domain again."

book

Article ID: 383255

calendar_today

Updated On:

Products

VMware vSAN 7.x

Issue/Introduction

This article explains the File Service - Infrastructure health in the vSAN Health Service and provides details on why it might report an error.

The vSAN File Service creation fails with error below on vCenter Web-Client




  • Error will be mentioned in the vCenter Server EAM :  /var/log/vmware/eam/eam.log

    2024-11-28T05:46:48.397Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
    com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.                                                                                 
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_402]
            at java.lang.Thread.run(Thread.java:750) [?:1.8.0_402]
    2024-11-28T05:46:53.760Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | 
    [Retry:Login:com.vmware.vim.eam:db634c32d91f3999] Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null   
    2024-11-28T05:46:53.761Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:c346af29820e1dfe] created from [Retry:Login:com.vmware.vim.eam:db634c32d91f3999]
    2024-11-28T05:46:56.768Z |  INFO | vim-async-1 | OpIdLogger.java | 43 | [vim:loginExtensionByCertificate:c346af29820e1dfe] Failed.
    2024-11-28T05:46:56.768Z |  WARN | vim-async-1 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:db634c32d91f3999] Re-login failed, due to:                                        
    com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.                                                                                               
            at com.vmware.eam.vim.security.impl.SessionManager.convertLoginException(SessionManager.java:295) ~[eam-server.jar:?]      

    This will repeat frequently at normal intervals, along with other authentication related errors in the eam.log. 

  • Error will be mentioned in the vCenter Server vSAN Health : /var/log/vmware/vsan-health/vmware-vsan-health-service.log 

    2024-11-28T01:52:33.152-06:00 INFO vsan-mgmt[11852] [VsanClusterFileServiceSystemImpl::_queryFileServiceConfigsImpl opID=noOpId] Calling host dalnetmod01b.mgmt.net.ibm.com for query file service config ...   >> Initialization of the vSAN File Service
    2024-11-28T01:52:33.163-06:00 ERROR vsan-mgmt[11852] [VsanClusterFileServiceSystemImpl::_queryFileServiceConfigsImpl opID=noOpId] Query file service config failed
    Traceback (most recent call last):
      File "bora/vsan/fileservice/vpxd/VsanClusterFileServiceSystemImpl.py", line 2099, in _queryFileServiceConfigsImpl
      File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
        self.f(*(self.args + (obj,) + args), **kwargs)
      File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
        return self._stub.InvokeMethod(self, info, args)
    PyCppVmomi.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
      msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00005635ebf9bd28, h:75, <TCP '9.46.37.8 : 53352'>, <TCP '9.46.37.74 : 443'>>), /vsan>]: queryFileServiceConfigs\nvim.fault.InvalidState",        >> File Service creation protocol failed 
      reason = 'Invalid fault'
    }

 

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

The EAM agent extension thumbprints may be mismatched. 

 

Resolution

Verify and Resolve the mismatched EAM extension thumbprint using one of the following methods:

Prechecks for vCenter Server Appliance (VCSA):

  1. Backup Preparation:

    • Ensure a full backup of the vCenter Server Appliance (VCSA) is taken in advance.
    • Validate the integrity of the backup to avoid complications during restoration, if required.

  2. Snapshot Creation:

    • Create a snapshot of the vCenter Server Appliance.
    • Ensure the snapshot includes all required states (e.g., memory and disk), and retain it until the operation is confirmed successful.

  3. Certificate Verification:

    • Confirm if the environment is configured with CA-signed certificates or external certificates.
    • If applicable, ensure you have access to the certificates and private keys for reconfiguration, if necessary.

Method 1: Manual thumbprint extension update (using vCenter CLI)

On the vCenter Server Appliance:

Log in to the vCenter Server Appliance using SSH.

    • Run this command to enable access the Bash shell :
      • shell.set --enabled true
    • Type shell and press Enter.


Run these commands to retrieve the vpxd-extension solution user certificate and key: 

    • mkdir /certificate
    • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
    • /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key 

First determine the vCenter's PNID. 

    •  /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Run this command to update the extension's certificate with vCenter Server (using the PNID after after the -s)

    • python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u [email protected]

Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment. When prompted, type in the [email protected] password.

Restart the VMware ESX Manager service with these commands:

    • service-control --stop vmware-eam
    • service-control --start vmware-eam

Method 2: The "Fixcerts" script can be used to replace certificates on the vCenter Server Appliance

To download the "Fixcerts" script Link : How to replace certificates on vCenter server using the Fixcerts script  

On the vCenter Server Appliance:

Log in to the vCenter Server Appliance using SSH.

      • Run this command to enable access the Bash shell :
        • shell.set --enabled true
      • Type shell and press Enter.

Update thumbprint for vpxd extensions eam :

      • python fixcerts_3_1.py update --ExtensionType eam

 

Powered by Wolken Software