Virtual machine snapshot and backup operations fail on a 2-node vSAN cluster.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Virtual machine backups fail on the 2-Node vSAN cluster.

• Virtual machine fails to create snapshots, with below error

  Error - Create virtual machine snapshot - Failed to place witnesses. There are currently 0 usable disks and 1 more usable disks are needed in witness node. An error occurred while taking a snapshot: 22 (Invalid argument).

Environment

VMware vSAN 8.x

Cause

• The VM snapshot and backup failures occurred because the vSAN witness node was in maintenance mode.

  

• From the logs of witness node -  /var/run/log/hostd.log

  ####-##-##T##:##:##.969Z In(166) Hostd[264596]: [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    key = 222,
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    chainId = -1017928264,
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    createdTime = "####-##-##T00:00:00Z",
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    userName = "",
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    host = (vim.event.HostEventArgument) {
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->       name = "WitessNode_HostName",
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->       host = 'vim.HostSystem:ha-host'
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    },
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    eventTypeId = "esx.audit.maintenancemode.entered",
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    objectId = "ha-host",
####-##-##T##:##:##.969Z In(166) Hostd[264387]: -->    objectType = "vim.HostSystem",
####-##-##T##:##:##.969Z In(166) Hostd[264387]: --> }
####-##-##T##:##:##.969Z In(166) Hostd[264596]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 91204 : The host has entered maintenance mode.

Also, from /var/run/log/vobd.log

####-##-##T##:##:##.968Z In(14) vobd[262422]:  [UserLevelCorrelator] 49396864497181us: [esx.audit.maintenancemode.entering] The host has begun entering maintenance mode.
####-##-##T##:##:##.075Z In(14) vobd[262422]:  [GenericCorrelator] 49396864603932us: [vob.user.maintenancemode.entered] The host has entered maintenance mode

• In this state, the disks on the witness appliance cannot accept new writes, which causes witness component creation to fail. Consequently, the system is prevented from creating any new vSAN objects, such as those required for snapshots.

• All objects in the vSAN cluster are in the reduced availability with no rebuild state as the witness node is in maintenance mode.

• Provisioning new vSAN objects, such as those required during snapshot creation, is not possible in the current cluster state. The applied vSAN RAID 1 storage policy dictates that the 2-Node cluster must have both data nodes and the witness available to fulfill the placement requirements for any new objects.

• As a result, backups fail because it is not possible to create the required snapshots.

Resolution

Exit maintenance mode on the vSAN witness node.