Verifying ESX/ESXi virtual machine file integrity
search cancel

Verifying ESX/ESXi virtual machine file integrity

book

Article ID: 311009

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article describes the various files that make up a virtual machine running on an ESX host. It also provides information on how the integrity of these files may be compromised and the various means of resolving the resulting problems.


Symptoms:
  • Unable to power on the virtual machine
  • Powering on the virtual machine fails
  • You see one of these errors:
    • A file was not found.
    • Operation not allowed in the current state

  • The <vmname>.vmdk file does not exist, but the <vmname-flat>.vmdk file exists.
  • Deleting snapshot fails with the error:

    ASSERT /build/mts/release/bora-39823/bora/vmx/snapshot/snapshot_vmx.c:752

  • VMFS volume becomes inaccessible due to file system corruption


Environment

VMware vSphere ESXi 5.0
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Installable
VMware vCenter Server 5.0.x
VMware ESXi 4.0.x Installable
VMware vCenter Server 4.1.x
VMware ESXi 3.5.x Installable
VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware VirtualCenter 2.5.x
VMware vCenter Server 4.0.x
VMware ESX Server 3.5.x
VMware ESXi 3.5.x Embedded

Resolution

Types of files

Before ensuring virtual machine file integrity, be aware of the various files that make up a virtual machine running on an ESX host.

VMX File

The VMX file is the primary configuration file of a virtual machine. Every aspect of your virtual machine is detailed in the VMX file, and any virtual hardware assigned to your virtual machine is present here. Here is an example entry for a virtual floppy drive:

floppy0.startConnected = "false"
floppy0.clientDevice = "true"
 
Each time you create a new virtual machine using the New Virtual Machine Wizard, the VMX file is appended to with each question you answer regarding the guest operating system, disk sizes, and networking.

VMXF File

This is a supplemental configuration file for virtual machines.

VMDK and -flat.VMDK Files

The <vm_name>-flat.vmdk files store the content of the virtual machines actual hard disk drives. Earlier versions of ESX used the extension .dsk for the virtual disk files.
 
The <vm_name>.vmdk files are the disk descriptor files. This file contains all the information about the associated flat file.
 

VMDK Delta Files

The <vm_name>-######.vmdk files store the content of the virtual machines disk from the time the snapshot was created.
 
The <vm_name>-######-delta.vmdk files are the disk descriptor files. This file contains all the information about the associated delta flat file.
 
The files tell you the CID of the file, the parent CID of the file, and the type and size of the file amongst other things.
 
This information can be very useful depending on what type of issue you are troubleshooting. It is especially useful when troubleshooting problems with snapshots.
 

VMSD File

This is a centralized file for storing information and metadata about a snapshots of a virtual machine.

 

VMSN File

This is the file which stores the state of a snapshot.

 

NVRAM File

This file contains the BIOS of the virtual machine.

Log Files

The vmware(-x).log files keep a log of key virtual machine activity. This file can be useful in troubleshooting if you encounter problems.

Renaming a virtual machine

Renaming your virtual machine can create inconsistencies that can cause issues.
 
To correctly rename a virtual machine:
 
Note: Avoid spaces, brackets, or non UTF-8 characters when naming virtual machines and their associated files.
  1. Commit any snapshots.
  2. Shutdown the virtual machine.
  3. If you are using vCenter Server, remove the virtual machine from the inventory but do not delete the files from disk.
  4. Connect to the ESX host on which the virtual machine resides over SSH. For more information, see Opening a command or shell prompt (1003892).
  5. Unregister the virtual machine.
    • For ESX, run this command:

      vmware-cmd -s unregister <path to config file>

      Where <path to config file> is the path to the configuration file as determined by the vmware-cmd –l command. 

      For example:

      vmware-cmd -s unregister '/vmfs/volumes/storage1/vm1/vm1.vmx'

    • For ESXi:
      1. Log in to the ESXi host where the virtual machine is registered. 
      2. Unregister the virtual machine with the CLI commands:

        Note: The getallvms command should return the vmid of the virtual machine to be unregistered.

        # vim-cmd vmsvc/getallvms
        # vim-cmd vmsvc/unregister vmid

        Alternatively, unregister the virtual machine from vCenter Server inventory. To unregister, right-click the virtual machine and choose Remove from Inventory.

  6. Rename the folder, .vmx file, and .vmdk (+ flat) file to match the new name:

    mv     /vmfs/volumes/storage1/vm1/ /vmfs/volumes/<perhaps new storage?>/NewVM/
    cd /vmfs/volumes/<perhaps new storage?>/NewVM/
    mv vm1.vmx NewVM.vmx
    mv vm1.vmdk NewVM.vmdk
    mv vm1-flat.vmdk NewVM-flat.vmdk

    See Note below for alternative command for renaming the VMDK file: vmkfstools -E <oldName>.vmdk <newName>.vmdk

  7. Edit the .vmx file to reflect the name of the new descriptor file.

    Locate the scsi0:0.fileName line. It appears similar to:

    scsi0:0.fileName = "vm1.vmdk"

    The edit appears similar to:

    scsi0:0.fileName = "NewVM.vmdk"

  8. Save the file and exit.
  9. Edit the .vmdk file to reflect the name of the new flat file.

    Locate the Extent description section of the .vmdk file. It appears similar to:

    # Extent description
    RW 8388608 VMFS "vm1-flat.vmdk"

    Edit this to:

    # Extent description
    RW 8388608 VMFS "NewVM-flat.vmdk"

  10. Register the virtual machine.
  11. Add the virtual machine back into vCenter Server inventory using the vSphere/VMware Infrastructure Client by browsing the data store, finding the .vmx file, right-clicking and adding it to inventory.
Note: There are some additional files that you may want to rename such as the .vmxf and the swap file name. Both of these are referenced from the .vmx file. If you decide to change either of these file names, ensure you update the .vmx file while you are completing step 7 in the instructions.
 
To rename a VMDK file, you can also use the command:

vmkfstools -E <oldName>.vmdk <newName>.vmdk

This command renames the content file (-flat.vmdk ) and also all references inside the metadata file (.vmdk). Ensure to update your virtual machines .vmx file after completing this step.

Corrupt VMX File

Issues with a virtual machine's .vmx file can prevent it from being powered on. You may also see these errors:
  • Error when deleting a snapshot: ASSERT /build/mts/release/bora-39823/bora/vmx/snapshot/snapshot_vmx.c:752
  • Error when attempting to power on a virtual machine: "operation not allowed in the current state"
For more information on .vmx files, see Validating the .vmx settings of a virtual machine (1003748).
 
If you suspect your .vmx file is preventing your virtual machine from starting, you can rebuild your virtual machine with a clean .vmx file.
Note: If the VM was previously in snapshot mode, be aware before adding disks to the New VM that you are reattaching the disks from the correct delta file. The following sample command grep -i scsi.*file *.vmx can be used to query the original .vmx file (if possible) outputting the delta file that the VM was last running from.
To rebuild a virtual machine:
  1. Connect to vCenter Server or your ESX host directly with the vSphere Client.
  2. Remove the virtual machine from the inventory.
  3. Right-click your ESX host and choose New Virtual Machine.
  4. Select the Custom option.
  5. Create the virtual machine with the same configuration it had before the migration.
  6. When you get to the Select a Disk section, choose to Use an existing virtual disk, and point to the original .vmdk file(s).
  7. Finish the creation of the virtual machine.
  8. Power on the virtual machine.
If your .vmx file has become corrupted, or the contents are missing, you can also rebuild the .vmx file if you still have the vmware.log file from the last time it booted correctly. For more information see Rebuilding the virtual machine's .vmx file from vmware.log (1023880).
 
When viewing the log file of the virtual machine, search for the '--- CONFIGURATION' section. The .vmx files is below.
 
You can copy this text to a text editor to create a working configuration file.
 
Note: Do not use an editor that inserts formatting that ESX does not understand (such as Wordpad). If possible, complete the editing on the ESX host.

The entries in the log file appear similar to:
 
Feb 28 18:11:47.799: vmx| DICT config.version = 8
Feb 28 18:11:47.799: vmx| DICT virtualHW.version = 4
Feb 28 18:11:47.799: vmx| DICT floppy0.present = TRUE
 
Remove the preceding section so it becomes:
 
config.version = 8
virtualHW.version = 4
floppy0.present = TRUE
 
Save it with the same name as the corrupted file and the replace the corrupted version.

Corrupt VMDK File

Issues with the VMDK file of a virtual machine can prevent it from being powered on. You may see the error:
 
Failed to power on a file was not found
 
This error may occur when the VMFS3 virtual disk descriptor (.vmdk ) files are missing. There are no <vmname>.vmdk files. Only <vmname-flat>.vmdk files exist.
 
You can also see this error if the Extent description section of the .vmdk file is incorrect. The log files show messages similar to:
 
Unable to find file
The system cannot find the file specified.
 
For more information on re-creating a missing descriptor file, see Recreating a missing VMDK (header) file (1002511).

VMFS Corruption

When a VMFS volume becomes inaccessible due to file system corruption, you may be able to migrate your running virtual machines off the filesystem. Powered off virtual machines are inaccessible.
 
Note: This process works if the disk blocks used for the virtual disks haven't been corrupted and the operating system is Microsoft Windows.
To migrate your virtual machine:
  1. Ensure you have sufficient disk space on another VMFS volume to clone the virtual machine.
  2. Connect to the virtual machine using remote desktop.
  3. Download and install VMware converter inside the virtual machine.

    This can also be downloaded to your own laptop, desktop, or another virtual machine as long as you have network access to both the source and destination.

  4. Run Converter and select Physical Machine and Local Machine.
  5. Choose an ESX Server host as a destination and select a healthy VMFS.
  6. Follow the wizard and wait for the process to complete.
Note: Do not power-off or restart the virtual machine. When powered-off, the virtual machine cannot start again as the files become inaccessible.
 
If the operating system is not Microsoft Windows, you can use native system tools to copy the whole image to a network share.
 
For example, on Linux:

# dd if=/dev/sda of=/mnt/nfs/sda.dd bs=1024k

You must create a new virtual machine and boot it from CD to Linux rescue mode and restore the image:


# dd if=/mnt/nfs/sda.dd of=/dev/sda bs=1024k

Note: This method creates only crash consistent image.
 
 



Additional Information

Powered by Wolken Software