Greenfield Workload Domain deployment fails at 'Checking the datastore availability' with 'FAILED_TO_CREATE_FILE_INSIDE_DATASTORE'
search cancel

Greenfield Workload Domain deployment fails at 'Checking the datastore availability' with 'FAILED_TO_CREATE_FILE_INSIDE_DATASTORE'

book

Article ID: 432964

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • SDDC Manager UI displays error: Failed to create file inside datastore <DATASTORE_ID> of datacenter <DATACENTER_ID>

  • Manual attempts to create folders or files on the vSAN datastore via the vSphere Client also fail.

  • The following log entries are captured from /var/log/vmware/vcf/domainmanager/domainmanager.log, indicating a failure during the datastore validation phase:
    YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-14]  Successfully logged in to https://<vcenter-fqdn>:443/sdk
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Getting entity type ClusterComputeResource with moid domain-## from vcenter
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Retrieved MoRef: {"_type":"ClusterComputeResource","_value":"domain-c##"}
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.s.s.a.DatastoreAvailabilityValidationAction,dm-exec-14]  datastore is accessible.Trying to create a test dir and delete it
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Getting entity type Datastore with moid datastore-id from vcenter
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Retrieved MoRef: {"_type":"Datastore","_value":"datastore-id"}
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Create new directory vcf-test-#### inside the datastore datastore-name in datacenter datacenter-id.createParentDirectories ? true
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Getting entity type Datacenter with moid datacenter-id from vcenter
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  Retrieved MoRef: {"_type":"Datacenter","_value":"datacenter-id"}
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,69a7#################0310,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-14]  FileFault exception while making a new Directory:com.vmware.vim.binding.vim.fault.CannotCreateFile: Cannot complete file creation operation.
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,69a7#################7f60,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-17]  [#####] FAILED_TO_CREATE_FILE_INSIDE_DATASTORE Failed to create file inside datastore datastore-id of datacenter datacenter-id

Environment

VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

The system is unable to perform I/O operations on the vSAN datastore due to network misconfiguration. Specifically, inconsistencies in the Network Pool assigned during ESXi host commissioning (e.g., incorrect VLANs, MTU mismatches, or invalid IP addresses for vSAN VMkernel interfaces) prevent the hosts from establishing the necessary storage quorum or communication required to commit file system changes to the vSAN object store.

Resolution

Note: The following procedure is applicable exclusively to greenfield deployment.

  1. Delete the Failed Workload Domain:

    • Log in to the SDDC Manager Dashboard.

    • Navigate to Inventory > Workload Domains.

    • Select the failed Workload Domain and select Actions > Delete.

  2. Decommission Affected ESXi Hosts:

    • Navigate to Inventory > Hosts..

    • Unassigned Hosts > Decommission Selected Hosts.

  3. Re-image the ESXi Hosts:

    • Perform a fresh installation of the ESXi version specified in the VCF Bill of Materials (BOM).

    • Note: This is necessary because manual changes to VMkernel interfaces or Network Pools post-commissioning often lead to sync failures between SDDC Manager, vCenter, and NSX.

  4. Verify Network Configuration:

    • Ensure the Network Pool in SDDC Manager has the correct VLAN, MTU (typically 9000 for vSAN), and Gateway for the vSAN and vMotion traffic.

    • Validate that the physical switch ports are trunked correctly for these VLANs.

  5. Re-commission and Retry:

    • Commission the freshly imaged hosts back into SDDC Manager using the corrected Network Pool.

    • Restart the Workload Domain deployment.

Powered by Wolken Software