vSAN File Service enablement fails: "Cannot create domain for the vSAN file service: Failed to check domain configuration"

Products

• VMware vSAN

Issue/Introduction

You are attempting to configure vSAN File Services on a vSAN cluster and the configuration wizard fails to complete the domain creation task. You observe the following error messages during this operation:

• "VDFS datastore is not present"
• "handshakeVer: failed to locate peer's certificate thumbprint"
• "The operation is not allowed in the current state. Cannot create domain for the vSAN file service: Failed to check domain configuration: (vmodl.fault.SystemError) { msg = "Received SOAP response fault from [<, >, /hgw/host-XXXXXX/vsan>]: performDomainConfigAdvancedCheck\nVDFS datastore is not present", reason = "VsanInvalidState('VDFS datastore is not present')" }"

Environment

vSAN FS 8.0 U3+

Cause

You experience this issue because the vSAN File Service (VDFS) proxy incorrectly handles chained certificates when Data-In-Transit encryption is enabled.

When your vCenter uses a Subordinate VMCA, the host presents a full certificate chain during the TLS handshake. The proxy attempts to verify the thumbprint for every certificate level in the chain (depth-0, depth-1, and depth-2); however, the host's configuration store only contains the thumbprint for the leaf certificate (depth-0). Because the deeper depth thumbprints are missing, the verification fails, aborting the vSAN File Service enablement.

For information regarding certificate-based failures in different environments, see vSAN File Service enablement fails at 20%

Resolution

There is currently no permanent fix today for this software limitation in vCenter Server and ESXi 8.0.3 patch 9 or below builds. A resolution is planned for vSphere 9.2 and vSphere 8.0 Patch 10.

To enable vSAN File Services in an environment utilizing a Subordinate VMCA with Data-In-Transit (DIT) encryption, you must use the following workaround:

1. Navigate to the vSphere Client.
2. Select your vSAN Cluster and navigate to Configure > vSAN > Services.
3. Locate the Data Protection section and click Edit.
4. Disable Data-In-Transit encryption by toggling the setting to Off and click Apply.
5. Wait for the "Update vSAN configuration" task to complete across all hosts.
6. Navigate to vSAN > Services > File Service and click Enable.
7. Once released, apply the patches containing the fix and re-enable Data-In-Transit encryption.

Additional Information