Failure in recovering of vSAN Cluster after manual shutdown using reboot_helper.py script

Products

• VMware vSAN

Issue/Introduction

• The vSAN cluster fails to restart after a manual shutdown using the prepare script documented in Manually Shut Down and Restart the vSAN Cluster
  
  

• Recovering cluster fails with Network error.
  
  [root@:~] python /usr/lib/vmware/vsan/bin/reboot_helper.py recover
  Begin to recover the cluster...
  Time among connected hosts are synchronized.
  Scheduled vSAN cluster restore task.
  Waiting for the scheduled task...(34s left)
  Waiting for the scheduled task...(4s left)
  Checking network status...
  Recovery is not ready, retry after 10s...
  Recovery is not ready, retry after 10s...
  Recovery is not ready, retry after 10s...
  Timeout, please try again later

Environment

VMware VSAN 7.x

VMware VSAN 8.x

Cause

Connectivity failed due to an MTU mismatch between the data and witness nodes.

Data nodes were set to 9000 MTU while the Witness was set to 1500 MTU.

Resolution

Please ensure the MTU is consistent across all nodes, then restart the management services on the ESXi hosts for the changes to take effect.

Re-run the recover script.

Once the script completes successfully, continue with the steps documented in Manually Shut Down and Restart the vSAN Cluster

Additional Information

Unable to restart a vSAN Cluster after manual shutdown using reboot_helper.py script

https://knowledge.broadcom.com/external/article?articleNumber=390160