vSAN File Service fails with Error " The File service is not available, because the file service domain is not successfully created. Click the edit button to create the file service domain again."

Products

VMware vSAN 7.x

Issue/Introduction

This article explains the File Service - Infrastructure health in the vSAN Health Service and provides details on why it might report an error.

The vSAN File Service creation fails with error below on vCenter Web-Client

The related error will be logged in the vCenter Server EAM log located at : /var/log/vmware/eam/eam.log

YYYY-MM-DDTHH:MM:SS. | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_402]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_402]
YYYY-MM-DDTHH:MM:SS. | INFO | vim-monitor | ExtensionSessionRenewer.java | 190 |
[Retry:Login:com.vmware.vim.eam:XXXXXXXXXXXXX] Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
YYYY-MM-DDTHH:MM:SS. | INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:####] created from [Retry:Login:com.vmware.vim.eam:######]
YYYY-MM-DDTHH:MM:SS. | INFO | vim-async-1 | OpIdLogger.java | 43 | [vim:loginExtensionByCertificate:####] Failed.
YYYY-MM-DDTHH:MM:SS. | WARN | vim-async-1 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:#######] Re-login failed, due to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
at com.vmware.eam.vim.security.impl.SessionManager.convertLoginException(SessionManager.java:295) ~[eam-server.jar:?]   This will repeat frequently at normal intervals, along with other authentication related errors in the eam.log.

The error will also be logged in the vCenter Server vSAN Health Service log located at: : /var/log/vmware/vsan-health/vmware-vsan-health-service.log

YYYY-MM-DDTHH:MM:SS. INFO vsan-mgmt[11852] [VsanClusterFileServiceSystemImpl::_queryFileServiceConfigsImpl opID=noOpId] Calling host ####.###.###for query file service config ... >> Initialization of the vSAN File Service
YYYY-MM-DDTHH:MM:SS. ERROR vsan-mgmt[11852] [VsanClusterFileServiceSystemImpl::_queryFileServiceConfigsImpl opID=noOpId] Query file service config failed
Traceback (most recent call last):
File "bora/vsan/fileservice/vpxd/VsanClusterFileServiceSystemImpl.py", line 2099, in _queryFileServiceConfigsImpl
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
PyCppVmomi.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00005XXXXXXXXX8, h:75, <TCP '##.##.##.## : 53352'>, <TCP '##.##.##.## : 443'>>), /vsan>]: queryFileServiceConfigs\nvim.fault.InvalidState", >> File Service creation protocol failed
reason = 'Invalid fault'
}

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

The EAM agent extension thumbprint mismatch is causing the vSAN File Service to become unavailable.
This happens because the File Service Domain fails to get created due to the thumbprint mismatch between the EAM extension and the vpxd-extension certificate.

Resolution

To Validate EAM Thumbprint Mismatch with vpxd-extension Certificate

Run the following commands to compare the thumbprint mismatch between the EAM extension and the vpxd-extension certificate:

Both outputs should display the same thumbprint. If they do not match, it indicates there is a thumbprint mismatch between the EAM extension and the vpxd-extension certificate.

In such a case, proceed with either of the following methods to update the EAM extension thumbprint to match the thumbprint of the vpxd-extension certificate:

Note: Before proceeding with the steps below, please ensure that a valid backup of the vCenter Server is taken. Additionally, take a snapshot of the vCenter.
If the vCenter is part of an Enhanced Linked Mode (ELM) configuration, it is strongly recommended to take an offline snapshot of all vCenter Servers that are part of the linked environment.

Method 1: Using "vCert" script to update the EAM thumbprint on the vCenter Server Appliance

To download the "vCert" script Link : vCert - expired certificate replacement script

Once the tool is downloaded transfer the file to VCSA appliance using winscp or any other available file transfer tool.

Then unzip the file use the below command to execute the file.

# unzip -q vCert-6.0.1-20250516.zip
# cd vCert-6.0.1-20250516
# chmod +x vCert.py
# ./vCert.py

2. When you execute the command you will be prompted with below question. Enter Y to continue.

Do you acknowledge the risks and wish to continue? [y/n]: y

3. Then enter option 3 to select manage certificates.

VCF Certificate Management Utility (version 6.0.1)
-----------------------------------------------------------------
1. Check current certificate status
2. View certificate info
3. Manage certificates
4. Manage SSL trust anchors
5. Check configurations
6. Reset all certificates with VMCA-signed certificates
7. ESXi certificate operations
8. Restart services
9. Generate certificate report
E. Exit

4. Then select option 6 to select vCenter extension thumbprint.

Manage vCenter Certificates
-----------------------------------------------------------------
1. Machine SSL certificate
2. Solution User certificates
3. CA certificates in VMware Directory
4. CA certificates in VECS Directory 5. SMS certificates
6. vCenter Extension thumbprints
7. STS signing certificates
8. VMCA certificate 9. Smart Card CA certificates 10. LDAPS Identity Source certificates 11. Clear expired certificates in BACKUP_STORE in VECS 12. Clear TRUSTED_ROOT_CRLS store in VECS 13. Clear Machine SSL CSR in VECS

If mismatches are detected, the wizard will prompt to update the extension thumbprints.

5. Then restart all the services on the VC either through vCert or you can try the below command.

service-control --stop --all && service-control --start --all

Method 2: Manual thumbprint extension update (using vCenter CLI)

On the vCenter Server Appliance:

Log in to the vCenter Server Appliance using SSH.

- Run this command to enable access the Bash shell :
  - shell.set --enabled true
- Type shell and press Enter.

Run these commands to retrieve the vpxd-extension solution user certificate and key:

- mkdir /certificate
- /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
- /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

Run the below command to capture the vCenter Server PNID.

- /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Run this command to update the extension's certificate with vCenter Server (using the PNID after after the -s)

- python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u <sso domain account>

Note: The default sso domain account is [email protected]. If this was changed during configuration, change the sso domain account to match your environment.

Restart the VMware ESX Manager service with these commands:

- service-control --stop vmware-eam
- service-control --start vmware-eam