ESX fails to boot when the disk containing the datastore with esxconsole.vmdk is detected as a snapshot

Products

• VMware

Issue/Introduction

Symptoms:
You may encounter issues while booting an ESX 4.x machine if:

• VMware ESX 4.x has been installed on the local disk or if it is booting from SAN
• There was a recent change to the disk or LUN where ESX 4.x has been installed
• There was a recent change to the controller connecting to the disk or LUN where ESX 4.x is installed
• The ESX host is attempting to boot from a LUN that has been replicated to another SAN.
• You see this issue in ESX hosts configured to boot from SAN and local disks
• When the VMFS volume is modified and needs a resignature, the ESX host fails to boot

You may also see these symptoms:

• The system does not boot and drops into Troubleshooting (busy box) mode.
• The boot process fails.
• You see the error:
  
  * vsd-mount .................................................... [Failed]

You have entered the recovery shell. The situation you are in may be recoverable.
If you are able to fix this situation the boot process will continue normally after you exit this terminal.
/Bin/SH: cant access TTY job control turned off

Cause

VMware ESX 4.x uses a virtual disk to store the service console. The virtual disk is created during the installation phase on a VMFS volume residing on the boot disk or LUN. The VMFS volume containing the service console files (also referred to as the esxconsole.vmdk) may be detected as a snapshot LUN. As a result of the VMFS volume being detected as a snapshot LUN, the ESX host may be unable to access the service console files during boot.

For more information about snapshot LUNs, see VMFS Volume Can Be Erroneously Recognized as a Snapshot (6482648).

Note: This article assumes that the disk, LUN, and storage controller are functioning properly. If you are having issues with local storage such as boot LUN presentation issues or a failed storage RAID, these issues must be identified before implementing the resolution provided in this article. Depending on the type of issue, you may need to engage the appropriate storage vendors for assistance.

Resolution

If the ESX host has detected the VMFS volume containing the esxconsole.vmdk file as a snapshot LUN, the ESX host drops into Troubleshooting (busy box) mode during boot.

Note: Depending on the installation type (for example, interactive mode or scripted mode), the service console files may be named differently. To verify the name of the service console file (esxconsole.vmdk, default-cos.vmdk, etc), run this command on the ESX command line:

# grep -i cosvmdk /etc/vmware/esx.conf

This procedure uses esxconsole.vmdk. Replace with the name of your service console file as necessary.

To allow your ESX host to boot successfully:
 

1. Provide the necessary credentials to access the busy box.
2. Run this command to enable automatic resignaturing of datastores on the host:
   
   # esxcfg-advcfg -s 1 /LVM/EnableResignature
   
   You must get output similar to:
   
   Value of EnableResignature is 1 .
   
   Note: If the root is mounted as read-only, run the mount -o remount / command to remount the volumes so that they are in a writable state.
    
3. Run this command to unload the VMFS drivers:
   
   # vmkload_mod -u vmfs3
    
4. Run this command to load the VMFS drivers:
   
   # vmkload_mod vmfs3
    
5. Run this command to detect new VMFS volumes and resignature the volume:
   
   # vmkfstools -V
    
6. Disable resignaturing on the host again (change its value back to 0):
   
   # esxcfg-advcfg -s 0 /LVM/EnableResignature
    
7. Run this command to identify the full path of the service console file:
   
   # find /vmfs/volumes/ -name esxconsole.vmdk
   
   The output appears similar to:
   
   /vmfs/volumes/4a14d968-88bf7161-700f-00145ef48f76/esxconsole-4a14d906-2f96-7956-7284-00145ef48f74/esxconsole.vmdk
   
   Notes:
   • The esxconsole.vmdk descriptor file is missing if no output is returned. To recreate the esxconsole.vmdk descriptor file, see Recreating a missing virtual machine disk (VMDK) descriptor file (1002511) and follow the steps to recreate the descriptor file, then continue with the steps above.
   • Make a note of this full path. After noting down the path information, you may skip step 7 and proceed with rebooting the ESX host.
8. Restart the VMware ESX machine. You see a menu provided by the grub boot loader.
9. Press e to edit the grub entries manually.
10. Scroll down to the line that starts with kernel /vmlinuz (it is indented under the VMware ESX 4.0 heading).
11. Press e (Edit key), go to the end of the line, and add a space followed by the option, all in one line, specifying the full path from step 7:
    
    /boot/cosvmdk=/vmfs/volumes/path/esxconsole.vmdk
     
12. Press Enter to accept the changes.
13. Press b to boot using the modified settings. The ESX host successfully boots.
    
    Note: The changes made to the boot options are not saved. They only apply to the current boot process. The changes need to be made to the boot configuration files as described in these steps:
     
    1. Log into the console as root.
    2. Edit the /etc/vmware/esx.conf file with a text editor and modify two lines, specifying the full path from step 7:
       
       /adv/Misc/CosCorefile = "/vmfs/volumes/path/core-dumps/cos-core"
/boot/cosvmdk = "/vmfs/volumes/path/esxconsole.vmdk"
        
    3. Run this command to update the boot configuration files:
       
       # esxcfg-boot -b

Additional Information

For more information regarding the grub boot loader, see the Grub's User Interface section of the Grub documentation.

Note:The preceding link was correct as of December 11, 2012. If you find the link is broken, provide feedback and a VMware employee will update the link.

For more information about snapshot LUNs, see VMFS Volume Can Be Erroneously Recognized as a Snapshot (6482648).

For more information about troubleshooting the esxconsole.vmdk file, see ESX host boot stops at the error: VSD mount/Bin/SH:cant access TTY job control turned off (1012874).
Recreating a missing virtual machine disk descriptor file
Snapshot LUN detection in ESX and ESXi
ESX host boot stops at the error: VSD mount/Bin/SH:cant access TTY job control turned off
esxconsole.vmdk を含むデータストアが格納されているディスクがスナップショットとして検出された場合に ESX を起動できない
ESX 在磁盘包含 esxconsole.vmdk 检测为快照的数据存储时无法引导
VMFS Volume Can Be Erroneously Recognized as a Snapshot