vSAN Cluster: All VMs Failed to Power On with Error “Failed to create swap file '/vmfs/volumes/vsan:… .vswp' : Failure”
search cancel

vSAN Cluster: All VMs Failed to Power On with Error “Failed to create swap file '/vmfs/volumes/vsan:… .vswp' : Failure”

book

Article ID: 419412

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSAN

Issue/Introduction

A healthy vSAN cluster may not be able to power on VMs with the error, “Failed to create swap file”, Unable create new VMs.

Module 'VMMON' power on failed.
Failed to create swap file '/vmfs/volumes/vsan:... .vswp' : Failure
Could not power on virtual machine: Failure
Failed to start the virtual machine.

swap: 3681: Failed to create swap file • 

The validation steps can be reviewed under the following scenarios:

1. vCenter VM is part of vSAN cluster :

Validation Steps : 

  • Unable to create New VM on vSAN LUN.
  • Unable to perform any VM operations.
  • VM is failed to Power on with below Error.

2. vCenter VM is not part of vSAN cluster:

  • New VMs or OVF files cannot be deployed to the specific vSAN cluster.
  • Existing VMs are unable to be migrated.
  • No operations are functioning within the cluster.
  • Unable to deploy VMs in vSAN cluster. Failed with an error " Cannot complete file creation operation. Failed to create an object".

 

/var/run/log/vpxa.log

YYY-MM-DD Er(163) Vpxa[2103872]: [Originator@6876 sub=Default opID=m2krkk2q-30393338-auto-i3fnf-h5:72537986-55-01-5f] [VpxLRO] -- ERROR lro-19899 -- 52403146-###-###-###-#####-- vpxa -- vpxapi.VpxaService.reserveName: :vim.fault.CannotCreateFile
YYY-MM-DD Er(163) Vpxa[2103833]: --> Result:
YYY-MM-DD Er(163) Vpxa[2103833]: --> (vim.fault.CannotCreateFile) {
YYY-MM-DD Er(163) Vpxa[2103833]: -->    faultCause = (vmodl.MethodFault) null,
YYY-MM-DD Er(163) Vpxa[2103833]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYY-MM-DD Er(163) Vpxa[2103833]: -->       (vmodl.LocalizableMessage) {
YYY-MM-DD Er(163) Vpxa[2103833]: -->          key = "vob.vsanprovider.object.creation.failed",
YYY-MM-DD Er(163) Vpxa[2103833]: -->          arg = <unset>,
YYY-MM-DD Er(163) Vpxa[2103833]: -->          message = "Failed to create object."
YYY-MM-DD Er(163) Vpxa[2103833]: -->       }
YYY-MM-DD Er(163) Vpxa[2103833]: -->    ],
YYY-MM-DD Er(163) Vpxa[2103833]: -->    file = "test (Cannot Create File)"
YYY-MM-DD Er(163) Vpxa[2103833]: -->    msg = "Received SOAP response fault from [<<io_obj p:0x000000517e7b7860, h:21, <TCP '127.0.0.1 : 34825'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: CreateDirectory
YYY-MM-DD Er(163) Vpxa[2103833]: --> Cannot complete file creation operation."
YYY-MM-DD Er(163) Vpxa[2103833]: --> }
YYY-MM-DD Er(163) Vpxa[2103833]: --> Args:
YYY-MM-DD Er(163) Vpxa[2103833]: -->
YYY-MM-DD Er(163) Vpxa[2103833]: --> Arg spec:
YYY-MM-DD Er(163) Vpxa[2103833]: --> (vpxapi.VmLayoutSpec) {
YYY-MM-DD Er(163) Vpxa[2103833]: -->    vmLocation = (vpxapi.VmLayoutSpec.Location) null,
YYY-MM-DD Er(163) Vpxa[2103833]: -->    multipleConfigs = <unset>,
YYY-MM-DD Er(163) Vpxa[2103833]: -->    basename = "test",
YYY-MM-DD Er(163) Vpxa[2103833]: -->    baseStorageProfile = "
YYY-MM-DD Er(163) Vpxa[2103833 Er(163) Vpxa[2103872]: [Originator@6876 sub=Default opID=m2XXXX-XXXX-XXXXX [VpxLRO] -- ERROR lro-19899 -- #####-#####-####-####-#####-- vpxa -- vpxapi.VpxaService.reserveName: :vim.fault.CannotCreateFile

Environment

vCenter Server

VMware vSAN

Cause

  • Validate vmkernel.log see any issues for state unsubscription.
  • Log Path: /var/run/log/vmkwarning.log 

YYY-MM-DD Wa(180) vmkwarning: cpu66:2100033)WARNING: DOM: DOMOwnerSubscribeExtNetInterfaceHealth:3865: Throttled: Either FSS not enabled or unsupported version for external net interface: 0
YYY-MM-DD In(182) vmkwarning: cpu66:2100033)DOM: DOMOwnerUnsubscribeClusterEncrState:6068: DOM Owner on 2aefd767-####-4733-#####-######## received premature cluster encryption state unsubscription

YYY-MM-DD In(182) vmkwarning: DOM: DOMOwnersubscribeExtNet1nterfaceHea1th:386S
YYY-MM-DD In(182) vmkernel: cpu50:2469440 opID=a2fdd9b2)World: 12745: VC opID m2krkk2q-30393338-auto-i3fnf-h5:####-55-01-5f-####-####maps to vmkernel opID a2fdd9b2
YYY-MM-DD Wa(180) vmkwarning: cpu50:2469440 opID=a2fdd9b2)WARNING: VSAN: VsanIoctlCtrlNodeCommon:3183: #####-0805-####-4484-#######: RPC to DOM op create returned: Failure 

This issue may occur if an incorrect vSAN license is applied, the required features are not supported by the current license, the evaluation license has expired, or the ESXi hosts in the cluster are not reflecting the licensed features associated with the vSAN license assigned to the cluster.

If vCenter is accessible click on the hamburger menu in the upper left, select Administration > Licensing > Licenses and check the vSAN licenses applied to the cluster.

If vCenter is not accessible as it's one of the VMs that is impacted check to see if the hosts reflect the appropriate licensed features per the vSAN license applied to the cluster on each host by running vsish -e get /config/VSAN/strOpts/LicensedFeatures

Here we see the "Current Value" is empty and no vSAN features are enabled.   

#  vsish -e get /config/VSAN/strOpts/LicensedFeatures

Vmkernel Config Option [string] {
   Default value:allflash,stretchedcluster
   valid characters:**
   Current value:
   hidden config option:1
   Description:List of licensed vSAN features for this host
   Host specific config option:0
   Option update requires reboot:0
   Option update requires maintenance mode:0
}

Expected output return a list of features enabled on the host per the license applied to the cluster such as in the example below. 

#  vsish -e get /config/VSAN/strOpts/LicensedFeatures
Vmkernel Config Option [string] {
   Default value:allflash,stretchedcluster
   valid characters:**
   Current value:vit,allflash,stretchedcluster,erasurecoding,storageefficiency,encryption
   hidden config option:1
   Description:List of licensed vSAN features for this host
   Host specific config option:0
   Option update requires reboot:0
   Option update requires maintenance mode:0

 

Resolution

Workaround Steps :

    1. Create a temporary test VM on the local datastore (not dependent on vSAN).

    2. Attach the vCenter VM’s vSAN disk to the temporary test VM by adding it as an existing disk. Add existing hard-disk to VM

    3. Power on the temporary test VM and verify the following:

      • Check all vCenter services by running:
        service-control --status --all

    4. Once access to vCenter is restored via the temporary test VM, proceed to apply the new  vSAN license through the vCenter UI.

    5. Power off the temporary test VM, restore the original configuration by reloading the original vCenter VM .vmx file, and then power on the original vCenter VM.

 

Additional Information

https://knowledge.broadcom.com/external/article/391524/unable-to-deploy-vms-in-vsan-cluster-wit.html

https://knowledge.broadcom.com/external/article/385058/vms-fail-to-power-on-with-error-failed-

 

Powered by Wolken Software