Failed to create object on newly added vSAN node in stretched cluster

Products

• VMware vSAN

Issue/Introduction

• Cluster is fully formed, new node is part of cluster, no cluster partition, vmkping to and from new node completes.
• Create virtual machine hangs at 6% for newly added host
• Create virtual machine reports "The operation is not allowed"
• Proactive test to create VM fails for newly added host 
• Object placement task in /var/log/clomd.log reports
  
  clomd[2103955] [Originator@6876 opID=1804289383] CLOMSetQuorumVotes: 1 upper primary witnesses are required
clomd[2103955] [Originator@6876 opID=1804289383] CLOMSetQuorumVotes: Need at least 3 Lower FDs Per Upper, current: 2
  
  

   

Environment

VMware vSAN 8.x

Cause

This newly added node to stretched cluster can ping the witness node but no communication over upd port 12321

Physical firewall was blocking udp port 12321 on this new node.

Resolution

Enable UDP port 12321 on firewall

Additional Information

1. Run "nc" to send UDP packet on port 12321 from the data node to the witness node:
   nc -u -z <witness-node-IP> 12321
2. Start packet capture on the witness node on the vmkernel port which is configured for vSAN traffic:
   tcpdump-uw -i <vmkX> -n udp port 12321
   Example: tcpdump-uw -i vmk1 -n udp port 12321 | grep <datanode-witness-vmkip>

 

Port requirements for vSAN: vSAN Ports and Protocols