General vSAN error. Failed to enter maintenance mode due to one or more candidate objects for migration being currently inaccessible. Retry operation in a different vSAN data migration mode.

Products

• VMware vSAN

Issue/Introduction

Symptoms :

• When attempting to put a vSAN Host in Maintenance Mode you will see the error:

  Error massage :---General vSAN error. Failed to enter maintenance mode due to one or more candidate objects for migration being currently inaccessible. Retry operation in a different vSAN data migration mode.

• The message appears on the  : vCenter UI client > Host and Cluster view >  ESXI Host  > Monitor > Tasks  > Task Name 

Validation steps:

• From the impacted node run the below commands on the impacted host and healthy host for network connectivity check. 

 # esxcli vsan network list
   Interface
   VmkNic Name: vmk2      >> vSAN dedicated VMkernel port
   IP Protocol: IPv4
   Interface UUID: efb99451-xxxx-xxxx-xxxx-005056011508
   Agent Group Multicast Address: 224.2.3.4
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: 224.1.2.3
   Master Group Multicast Port: 12345
   Multicast TTL: 5

 

• From the impacted node run the below commands on the impacted host and healthy host for inaccessible object check.
   
  
  • The impacted node will show objects as inaccessible 
    
    esxcli vsan debug object health summary get
Health Status                                              Number Of Objects
---------------------------------------------------------  -----------------
remoteAccessible                                                           0
inaccessible                                                               100
reduced-availability-with-no-rebuild                                       0
reduced-availability-with-no-rebuild-delay-timer                           0
reducedavailabilitywithpolicypending                                       0
reducedavailabilitywithpolicypendingfailed                                 0
reduced-availability-with-active-rebuild                                   0
reducedavailabilitywithpausedrebuild                                       0
data-move                                                                  0
nonavailability-related-reconfig                                           0
nonavailabilityrelatedincompliancewithpolicypending                        0
nonavailabilityrelatedincompliancewithpolicypendingfailed                  0
nonavailability-related-incompliance                                       0
nonavailabilityrelatedincompliancewithpausedrebuild                        0
healthy                                                                    0

Environment

vSAN 7.x

vSAN 8.x

Cause

• The error occurs because of the network partition of  ESXI host from rest of cluster

vmkping -I <vmk-VSAN> -d -s 1472 <IP address of the VMK vSAN of other node(s) in cluster>
PING <IP address>  (IP Address): 1472 data bytes
request timed out.
request timed out.
request timed out.
request timed out.

ping statistics for <IP Address> :
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Resolution

• Investigate the network on the in a vSAN cluster and ensure the ICMP connectivity  is good and the ports are open.
  
  
• The ports needed for the vSAN VMkernel are listed in the  (  https://ports.broadcom.com/home/vSAN ) 

Network Investigation in a vSAN Cluster:

1. Verify ICMP Connectivity:

   • Perform an ICMP ping test across all vSAN nodes to ensure network connectivity is stable and consistent.

   • Check for any packet loss or latency issues that could affect performance.

2. Ensure Proper Port Availability:

   • Confirm that all required ports for vSAN VMkernel are open and not being blocked by firewalls or other network security devices.

   • For the specific ports required for vSAN VMkernel traffic, refer to the official documentation  https://ports.broadcom.com/home/vSAN 

3. Network Health Check:

   • Check network configuration on each host to ensure they are correctly set up for vSAN, including VLAN settings, MTU (Jumbo Frames), and redundancy.

   • Validate the physical network devices (NICs, switches) to confirm they are configured and functioning properly.

Additional Information

• If the network connectivity is ok , then check for the article for inaccessible objects in the vSAN cluster :  vSAN -- Host Maintenance Mode fails with General vSAN error 
  
  
  • Use the VMkernel port ( vmk2 ) from above command, to ping the other hosts in cluster  ( Refer : Testing VMkernel network connectivity with the vmkping command )
    
    
    • The following commands are used for network ICMP check for normal and jumbo frames
       
      
      • Normal ping packet check

  vmkping -I vmk2 -d -s 1472 <IP address of the vSAN VMK of other node in cluster>
PING <IP address>  (IP Address): 1472 data bytes
1480 bytes from <IP Address> : icmp_seq=0 ttl=255 time=0.304 ms
1480 bytes from <IP Address> : icmp_seq=1 ttl=255 time=0.341 ms
1480 bytes from <IP Address> : icmp_seq=2 ttl=255 time=0.259 ms

  • • • Jumbo frames check

  vmkping -I vmk2 -d -s 8972 <IP address of the vSAN VMK of other node in cluster>
PING <IP address>  (IP Address): 1472 data bytes
1480 bytes from <IP Address> : icmp_seq=0 ttl=255 time=0.304 ms
1480 bytes from <IP Address> : icmp_seq=1 ttl=255 time=0.341 ms
1480 bytes from <IP Address> : icmp_seq=2 ttl=255 time=0.259 ms


  • • • Packet capture command 

  pktcap-uw --vmk vmk2 --dir 2 --dstip < Destination vSAN VMkernel IP > -o - | tcpdump-uw -ner -   ( capture the output in a file of the putty session ) 
  
  nc -uvz < Destination vSAN VMkernel IP > 12321
Connection to < Destination vSAN VMkernel IP > 12321 port [udp/*] succeeded!