Unable to create VM on some Datastores but not others.

Products

• VMware vCenter Server

Issue/Introduction

When using VAST Data storage with VMware ESXi over NFS, you experience the following symptoms:

 

• Virtual Machines (VMs) suddenly enter an "Inaccessible" or "Orphaned" state.

 

• Attempts to power on a VM fail with: An error occurred while challenging the device (13).

 

• The vmkernel.log on the ESXi host shows:

 

        NFS: 157: NFS write error Write failed with 13 (Permission denied)

 

• The VAST Management Console shows traffic originating from a VMkernel IP, but the file operations are being mapped to nfsnobody (UID 65534).

 

Environment

Esxi 7.x

Esxi 8.x

Esxi 9.x

Cause

This issue occurs when the VAST Cluster loses the mapping context for a specific VMkernel NFS IP address. Because the VAST View (Export) Policy typically restricts No Root Squash privileges to specific trusted IPs, any traffic that is not explicitly identified as "Trusted" is defaulted to Root Squashing.

 

This identity loss can be triggered by:

 

• IP/MAC Re-learning Delays: Network failover events where the VMkernel moves to a different physical uplink.

 

• VAST CNode Failover: A VAST CNode rebooting, causing the VIP (Virtual IP) to migrate before the ARP table is fully refreshed across the fabric.

 

• Subnet Ambiguity: Multiple VMkernel interfaces on the same subnet causing the host to egress traffic via an unintended interface.

 

Resolution

To resolve this issue, the connection state must be refreshed to ensure the VAST Cluster correctly identifies the source IP of the VMkernel.

 

1. Verify the Active Path

 

Log into the ESXi shell and identify which VMkernel interface is currently being used for the NFS mount:

 

Bash

#esxcli network ip connection list | grep 2049

 

Cross-reference the Local Address in the output with your VAST Export Policy "Allowed IPs" list.

 

2. Force ARP Refresh

 

If the IP is correct but still being squashed, force a refresh of the ARP entry on the ESXi host to re-announce itself to the VAST VIP:

 

Bash

 

# Replace [Gateway_IP] with the gateway for the NFS subnet

 

#arp -d [Gateway_IP]

 

#ping -I vmkX [VAST_VIP_IP]

 

3. Adjust VAST View Policy (Workaround)

 

If the VMkernel IPs are dynamic or prone to shifting, consider broadening the "No Root Squash" rule:

 

1. Navigate to VAST Management Console > Element Store > Views.

 

2. Edit the affected Policy.

 

3. Ensure the NFS State is set to No Root Squash for the entire subnet used by the VMkernels, rather than individual IPs.

 

Additional Information

Reach out to VAST support on this issue.