Replacing/Recreating Base or Snapshot Descriptor files for vSAN OSA Objects
search cancel

Replacing/Recreating Base or Snapshot Descriptor files for vSAN OSA Objects

book

Article ID: 326538

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

There are certain instances where descriptor files, both base disk & snapshots, may get damaged, corrupted, or deleted. This KB will outline the following:

  • Creating a new descriptor file both base disk and snapshot for vSAN OSA object vmdks
  • Properly formatting the descriptor file
  • Set attribute to the object path
  • Attach the new descriptor to the virtual machine's configuration file (.vmx)

THE CONTENT OF THIS ARTICLE IS PROVIDED "AS-IS," AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, VMWARE BY BROADCOM DISCLAIMS ALL OTHER REPRESENTATIONS AND WARRANTIES, EXPRESSED OR IMPLIED, REGARDING THIS CONTENT, INCLUDING THEIR FITNESS FOR A PARTICULAR PURPOSE, THEIR MERCHANTABILITY, OR THEIR NONINFRINGEMENT. VMWARE BY BROADCOM SHALL NOT BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS CONTENT, INCLUDING DIRECT, INDIRECT, CONSEQUENTIAL DAMAGES, LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF VMWARE BY BROADCOM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

This is for vSAN OSA ONLY and should NOT be used for vSAN ESA as Native Snapshots are handled differently and the descriptor files are not the same.

ENSURE TO READ THROUGH THE ENTIRE KB ENSURING YOU UNDERSTAND THE PROCESS BEFORE RECREATING MISSING OSA DESCRIPTOR FILES. IF YOU'RE UNSURE OF ANY STEPS OPEN A CASE WITH vSAN Support FOR ASSISTANCE.

Environment

VMware vSAN OSA (All Versions)

Resolution

  1. Log into two hosts of the cluster via SSH session. One will be for the host where the VM resides as mentioned in Step 2 and one for gathering the info required to recreate the descriptor file, Steps 3 & 4.

  2. Ensure you have a healthy namespace/directory for the VM available in order to create the new descriptor file(s).

    1. Log into the host that the VM is registered to and then run the following command:
      vim-cmd vmsvc/getallvms | grep <name of the VM> this should give you the following output.

      Example:
      [root@esxi_host:~] vim-cmd vmsvc/getallvms | grep Test-VM
      1      Test-VM   [vsanDatastore] 35873762-12ab-1ccc-acda-############/Test-VM.vmx

    2. Once you have the path to the namespace cd to that namespace - You will need to be in the namespace of the VM for the remaining steps

      Example:
      [root@esxi_host:~] cd /vmfs/volumes/vsanDatastore/35873762-12ab-1ccc-acda-############

    3. If the namespace doesn't exist create a new VM container with the exact settings used for the VM to create the missing namespace with no disks.

    4. If there is a namespace for the VM and you only need to recreate new descriptor file(s) proceed to step 3.

  3. Generate a list of the objects associated with the VM(s) by running the below commands:

    1. esxcli vsan debug object list --all (for 7.x) or --max-number=<number higher than the objects in the environment>(for 8.x as there is a bug where --all doesn't work) > /tmp/debugObjList.txt
      cat /tmp/debugObjList.txt  | grep "Object UUID\|Path" | awk '{printf $0}' | sed -e s"/Object UUID/\nObject UUID/g" | grep <vm-name> > /tmp/debug<vm-name>.txt

      Example:
      cat /tmp/debugObjList.txt  | grep "Object UUID\|Path" | awk '{printf $0}' | sed -e s"/Object UUID/\nObject UUID/g" | grep Test-VM > /tmp/debugTest-VM.txt

    2. cat /tmp/debugTest-VM.txt
      Object UUID: 7c883762-da93-2214-5a27-############   Path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM-000001.vmdk (Exists)
      Object UUID: 3c873762-b7d0-6c49-35dd-############   Path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM.vmdk (Exists)
      Object UUID: 35873762-12ab-1ccc-acda-############   Path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/Test-VM (Exists)

  4. Run /usr/lib/vmware/osfs/bin/objtool getAttr -u <object_UUID> |grep -E 'Object size|Object path' against the vSAN UUID object you need to create a new descriptor file for to ensure the Object attributes are referencing the proper vmdk descriptor path and to get the object size.

    Example:

    /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############] /usr/lib/vmware/osfs/bin/objtool getAttr -u 3c873762-b7d0-6c49-35dd-############ |grep -E 'Object size|Object path'

    Object size:17179869184 <this is the info you will need for step 5>

    Object path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM.vmdk
    (Please verify the object attribute (name) is the same as the object you need to create the new descriptor for)

  5. Copy the below Base disk descriptor template and paste it into Notepad or Notepad++

    # Disk DescriptorFile
    version=4
    encoding="UTF-8"
    CID=43d91c46
    parentCID=ffffffff
    createType="vsan"

    # Extent description
    RW <Object Size from Step 4 /512 here> VMFS "vsan://<vsan datastore UUID from Object Path info from step 4 here>/Object UUID for the object of the descriptor file being recreated from step 3 here>"

    #The Disk Data Base
    #DDB


  6. Fill in the respective object size, vSAN datastore UUID, Object UUID as indicated from the template

    Note: The object size from step 4 is in sectors so the value needs to be divided by 512 and then enter that value into the respective place for the descriptor file 

  7. Create a new file inside the namespace of the datastore named for the VMDK descriptor file being recreated.

    [root@esxi_host:/vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############] vi Test-VM.vmdk
    Note: Use the same file name as the Object Path info from Step 4

  8. Copy the completed template press the "Insert" key and then paste the template into the file

  9. Save the new descriptor file press "Esc" key then type :wq! then "Enter"

  10. Check the newly created descriptor file by using either vmkfstools -e or vmkfstools -qv10

    Example output:
    # vmkfstools -qv10 Test-VM.vmdk
    DISKLIB-VMFS : "vsan://523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############" : open successful (23) size = 17179869184, hd = 0. Type 3
    ./Test-VM.vmdk is not an rdm <== If you get here, you have passed the test!
    DISKLIB-VMFS : "vsan://523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############" : closed.
    AIOMGR-S : stat o=3 r=3 w=0 i=2 br=49152 bw=0
    OBJLIB-LIB: ObjLib cleanup done.
    WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

    OR:

    # vmkfstools -e Test-VM.vmdk
    Disk chain is consistent.

  11. Power on the virtual machine
    Note: If recreating descriptors for VMs containing snapshots follow the steps below before powering on the VM

  12. Copy the below snapshot descriptor template and paste it into Notepad or Notepad++

    # Disk DescriptorFile
    version=4
    encoding="UTF-8"
    CID=fffffffe
    parentCID=fffffffe
    createType="vsanSparse"
    parentFileNameHint="Test-VM.vmdk"

    # Extent description
    RW <Object Size from Step 4 /512 here> VMFS "vsan://<vsan datastore UUID from Object Path info from step 4 here>/Object UUID for the object of the descriptor file being recreated from step 3 here>"

    #The Disk Data Base
    #DDB

  13. Fill in the respective object size, vSAN datastore UUID, Object UUID as indicated from the template

    Note: The object size from step 4 is in sectors so the value needs to be divided by 512 and then enter that value into the respective place for the descriptor file 

  14. Create a snapshot/delta descriptor file vi <vmname>-000001.vmdk
    Where vmname = the name of the VM and the trailing -000001 is the number in the snapshot chain, where the format is always just (-) followed by 6 characters so snapshot 13 will look like -000013.

    The main things to consider when recreating descriptor files for VMs with snapshots and rebuilding the chain are the following:
    1. The sector's value after RW needs to be the same throughout all descriptor files for the snapshot chain
    2. The parentFileNameHint needs to reference the next disk in sequence for the chain ie: examplename-000002.vmdk may point to examplename-000001.vmdk and examplename-000001.vmdk may point to the base disk examplename.vmdk
    3. The UUID of the vSAN object the snapshot descriptor file points to needs to be updated to the corresponding vSAN object UUID
    4. The parentCID needs to point to the CID of the next descriptor in the sequence.

      Note: When rebuilding snapshot chain for multiple vmdks for a VM you can keep the same parentCID/CID relationship for all chains saving some time during the recreation process. Just ensure you're updating a,b, & c above accordingly for their respected disk chains

      Note: You can also completely build the templates out in Notepad or Notepad++ first then copy the completed templates into the new descriptor files being created. 


  15. Once all snapshot/delta descriptor files have been created ensure to follow step 9 for creating base disk descriptor files to vet the snapshot chain.

  16. If you had to create a new VM container prior to recreating the descriptor files you will need to set the new Object Path for the descriptor files by running command /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>

    Example:
    /usr/lib/vmware/osfs/bin/objtool setAttr -u 35873762-12ab-1ccc-acda-############ -d /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM.vmdk


  17. If you need assistance with this process open a case with vSAN Support

Additional Information

Another option is to use vmkfstools -c for recreating the base disk descriptor. This doesn't work for recreating snapshot/delta descriptor files. If you need to recreate snapshot descriptor files see the Resolution section above.

Below are the steps.

  1. Log into the host the VM resides via SSH session
  2. Navigate to the VM directory
  3. Generate a list of the objects associated with the VM(s) by running the below commands:

    1. esxcli vsan debug object list --all (for 7.x) or --max-number=<number higher than the objects in the environment>(for 8.x as there is a bug where --all doesn't work) > /tmp/debugObjList.txt
      cat /tmp/debugObjList.txt  | grep "Object UUID\|Path" | awk '{printf $0}' | sed -e s"/Object UUID/\nObject UUID/g" | grep <vm-name> > /tmp/debug<vm-name>.txt

      Example:
      cat /tmp/debugObjList.txt  | grep "Object UUID\|Path" | awk '{printf $0}' | sed -e s"/Object UUID/\nObject UUID/g" | grep Test-VM > /tmp/debugTest-VM.txt

    2. cat /tmp/debugTest-VM.txt
      Object UUID: 7c883762-da93-2214-5a27-############   Path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM-000001.vmdk (Exists)
      Object UUID: 3c873762-b7d0-6c49-35dd-############   Path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM.vmdk (Exists)
      Object UUID: 35873762-12ab-1ccc-acda-############   Path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/Test-VM (Exists)

  4. Run /usr/lib/vmware/osfs/bin/objtool getAttr -u <object_UUID> |grep -E 'Object size|Object path' against the vSAN UUID object you need to create a new descriptor file for to ensure the Object attributes are referencing the proper vmdk descriptor path and to get the object size.

    Example:

    /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############] /usr/lib/vmware/osfs/bin/objtool getAttr -u 3c873762-b7d0-6c49-35dd-############ |grep -E 'Object size|Object path'

    Object size:17179869184 <this is the info you will need for step 5>

    Object path: /vmfs/volumes/vsan:523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############/Test-VM.vmdk
    (Please verify the object attribute (name) is the same as the object you need to create the new descriptor for)

     

     

  5. Run command vmkfstools -c <Object size> VM_name.vmdk to create the new base disk descriptor file

    Example:
    vmkfstools -c 17179869184 Test-VM.vmdk

    Note: This command will create an object on the vSAN datastore as well which will need to be deleted later.

  6. Edit the the new descriptor file to  point to the correct base disk object.
    1. vi Test-VM.vmdk
    2. Copy the Object UUID listed in the line starting with RW this is the object that will need to be deleted in step 9
    3. Update the RW line ensuring the object size is the same as the size used in the command and update the object UUID to match the object the new descriptor was created for.

      Example: 
      # Extent description
      RW 17179869184  VMFS "vsan://523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############"
    4. Save the new descriptor file press "Esc" key then type :wq! then "Enter"

  7. Check the newly created descriptor file by using either vmkfstools -e or vmkfstools -qv10

    Example output:
    # vmkfstools -qv10 Test-VM.vmdk
    DISKLIB-VMFS : "vsan://523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############" : open successful (23) size = 17179869184, hd = 0. Type 3
    ./Test-VM.vmdk is not an rdm <== If you get here, you have passed the test!
    DISKLIB-VMFS : "vsan://523a45b14aa3d25e-3d06############/35873762-12ab-1ccc-acda-############" : closed.
    AIOMGR-S : stat o=3 r=3 w=0 i=2 br=49152 bw=0
    OBJLIB-LIB: ObjLib cleanup done.
    WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

    OR:

    # vmkfstools -e Test-VM.vmdk
    Disk chain is consistent.

  8. Power on the virtual machine

  9. Delete the object that was created when vmkfstools -c <Object size> VM_name.vmdk was initially run

    Run /usr/lib/vmware/osfs/bin/objtool delete -u <object_UUID> to delete the object
    Note: It's crucial to double triple check the correct uuid is used for this command cause once the object is deleted it's gone and will need to be restored from a backup

 

Powered by Wolken Software