VM goes inaccessible while placing witness node in maintenance mode.

Products

• VMware vSAN

Issue/Introduction

• In a 2 node vSAN cluster, Virtual machines are going inaccessible while taking the witness node into maintenance mode 
  
  
• In vCenter UI, Virtual Machine > Monitor > vSAN > Physical disk placement shows 2 Witness components for the object.

Environment

VMware vSAN (All Versions)

Cause

Virtual machine goes inaccessible as the object lose quorum as there are 2 witness components for the object with 1 vote each.

[root@ESXi01:~] esxcli vsan debug object list -u ########-####-####-####-###########

Object UUID: ########-####-####-####-###########

   Version: 10

   Health: inaccessible - Lost quorum.(APD)

   Owner: ESXi01

   Size: 0.00 GB

Configuration:
      RAID_1

         Component: ########-####-####-####-###########

           Component State: ACTIVE,  Address Space(B): 268435456000 (250.00GB),  Disk UUID: ########-####-####-####-###########,  Disk Name: naa.#######################:2

           Votes: 1,  Capacity Used(B): 135866089472 (126.54GB),  Physical Capacity Used(B): 134519717888 (125.28GB),  Host Name: ESXi#.local

         Component: ########-####-####-####-###########

           Component State: ACTIVE,  Address Space(B): 268435456000 (250.00GB),  Disk UUID: ########-####-####-####-###########,  Disk Name: naa.#######################:2

           Votes: 1,  Capacity Used(B): 135866089472 (126.54GB),  Physical Capacity Used(B): 134519717888 (125.28GB),  Host Name: #-ESXi#

      Witness: ########-####-####-####-###########

        Component State: ABSENT,  Address Space(B): 0 (0.00GB),  Disk UUID: ########-####-####-####-###########,  Disk Name: mpx.vmhba0:C0:T1:L0:2

        Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: ##-Witness.#.local

      Witness: ########-####-####-####-###########

        Component State: ABSENT,  Address Space(B): 0 (0.00GB),  Disk UUID: ########-####-####-####-###########,  Disk Name: mpx.vmhba0:C0:T1:L0:2

        Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: ##-Witness.#.local

Resolution

Workaround:

• Create a new FTT0 storage policy with below settings:
  Site disaster Tolerance: None - Standard cluster
  Failure to tolerate: No data redundancy 
  
  
• Apply the FTT0 storage policy to the VM and wait for the resync to complete.
  
  
• Change the storage policy back to the original storage policy.

Note: FTT0 policy will not provide any redundancy for the data. Any failure reported while the VM using the FTT0 policy may cause data loss. Hence recommended to have a backup of the Virtual machines before performing this workaround.