Moving vSAN VMDKS from one Namespace Folder to another does not update the Object Path Attribute
Article ID: 315539
Updated On:
Products
VMware vSAN
Issue/Introduction
The purpose of this article is to provide a detailed example of how this behavior can occur, along with corrective measures available using ESXi's vSAN Object Tool, or objtool.
Symptoms:
The output of esxcli vsan debug object list can give VMDKS that have been moved between Namespaces the appearance of being (Missing), thus making them seem Orphaned or Unassociated.
For example:
[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: ########-####-####-####-########3753
Version: 7
Health: healthy
Owner: vSAN-1.gsslabs.org
Size: 10.00 GB
Used: 0.01 GB
Policy:
CSN: 2
stripeWidth: 1
cacheReservation: 0
proportionalCapacity: 0
spbmProfileName: vSAN Default Storage Policy
forceProvisioning: 0
spbmProfileId: ########-####-####-####-########5bad
hostFailuresToTolerate: 1
spbmProfileGenerationNumber: 0
Type: vdisk
Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk (Missing)
Group UUID: ########-####-####-####-########3753
Directory Name: N/A
Symptoms:
The output of esxcli vsan debug object list can give VMDKS that have been moved between Namespaces the appearance of being (Missing), thus making them seem Orphaned or Unassociated.
For example:
[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: ########-####-####-####-########3753
Version: 7
Health: healthy
Owner: vSAN-1.gsslabs.org
Size: 10.00 GB
Used: 0.01 GB
Policy:
CSN: 2
stripeWidth: 1
cacheReservation: 0
proportionalCapacity: 0
spbmProfileName: vSAN Default Storage Policy
forceProvisioning: 0
spbmProfileId: ########-####-####-####-########5bad
hostFailuresToTolerate: 1
spbmProfileGenerationNumber: 0
Type: vdisk
Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk (Missing)
Group UUID: ########-####-####-####-########3753
Directory Name: N/A
Environment
VMware vSAN 6.x
VMware vSAN 7.0.x
VMware vSAN 5.5.x
VMware vSAN 8.0.x
VMware vSAN 7.0.x
VMware vSAN 5.5.x
VMware vSAN 8.0.x
Cause
When moving vSAN VMDK Descriptor Files from one Namespace directory to a different Namespace directory, the Object Path Attribute is not automatically updated.
By default, vSAN does not move VMDK Descriptor Files from one Namespace to a different Namespace.
This is expected product behavior. The Object Path Attribute of vSAN Objects must be updated explicitly with no automated methods currently available.
By default, vSAN does not move VMDK Descriptor Files from one Namespace to a different Namespace.
This is expected product behavior. The Object Path Attribute of vSAN Objects must be updated explicitly with no automated methods currently available.
Resolution
Currently there is no mechanism to prevent this behavior from happening as vSAN does not move VMDK Descriptor files between Namespace Directories. However, the need to move files between Namespaces should be a relatively rare scenario.
Workaround:
See step 5 in the Related Information section.
Workaround:
See step 5 in the Related Information section.
Additional Information
This behavior can be reproduced when VMDK Descriptor files are moved manually using the Datastore Browser in the vSphere Client UI, when using "mv" commands in ESXi's command line interface, and when VMDKS are moved using vSphere API's such as MoveDatastoreFile_Task.
In the example scenario below, we'll walk through the process of reproducing and correcting this behavior with two example Virtual Machines: "Virtual Machine 1" & "Virtual Machine 2":
"Virtual Machine 1" Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753
"Virtual Machine 2" Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753
1. First, we can obtain the Object UUID of "Virtual Machine 1.vmdk" by simply cat'ing the VMDK Descriptor file -
[root@vSAN-1:~] cat "/vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk"
# Disk DescriptorFile
version=4
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
createType="vmfs"
# Extent description
RW 20971520 VMFS "vsan://########-####-####-####-########3753"
# The Disk Data Base
#DDB
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "1305"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "0046fa2bcb13ce19e6ba05cffffffffe"
ddb.thinProvisioned = "1"
ddb.uuid = "60 00 C2 92 ## ## ## ##-## ## ## ## a2 fe af 7a"
ddb.virtualHWVersion = "14"
2. Next, we can move "Virtual Machine 1.vmdk" to Virtual Machine 2's Directory using a "mv" command:
[root@vSAN-1:~] mv "/vmfs/volumes/vsanDatastore/########-####-####-####-########3753/Virtual Machine 1.vmdk" "/vmfs/volumes/vsanDatastore/########-####-####-####-########3753"
3. Using the vSphere Client UI, we can attach "Virtual Machine 1.vmdk" to Virtual Machine 2:
4. Reviewing the output of esxcli vsan debug object list, we can now see that the VMDK Descriptor is reported as (Missing), even though it still exists:
[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: ########-####-####-####-########3753
Version: 7
Health: healthy
Owner: vSAN-1.gsslabs.org
Size: 10.00 GB
Used: 0.01 GB
Policy:
CSN: 2
stripeWidth: 1
cacheReservation: 0
proportionalCapacity: 0
spbmProfileName: vSAN Default Storage Policy
forceProvisioning: 0
spbmProfileId: ########-####-####-####-########5bad
hostFailuresToTolerate: 1
spbmProfileGenerationNumber: 0
Type: vdisk
Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk (Missing)
Group UUID: ########-####-####-####-########3753
Directory Name: N/A
This is because the Path Attribute for this Object still references the previous directory from Virtual Machine 1. This can be confirmed via an "objtool getAttr" (Object Tool Get Attribute) command:
* Command Syntax: /usr/lib/vmware/osfs/bin/objtool getAttr -u <Object UUID>
[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool getAttr -u ########-####-####-####-########3753
Object Attributes --
UUID:########-####-####-####-########3753
Object type:vsan
Object size:10737418240
User friendly name:(null)
HA metadata:(null)
Allocation type:Thin
Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"########-####-####-####-########5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))
Object class: vdisk
Object capabilities: NONE
Object path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk
Group uuid: ########-####-####-####-########3753
Container uuid: (null)
5. We must update the Path Attribute using an "objtool setAttr" (Object Tool Set Attribute) command.
* Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>
[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool setAttr -u ########-####-####-####-########3753 -d "/vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk"
Object set attribute succeeded
Re-running esxcli vsan debug object list, we can now confirm the VMDK Descriptor File is no longer reported as (Missing):
[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: ########-####-####-####-########3753
Version: 7
Health: healthy
Owner: vSAN-1.gsslabs.org
Size: 10.00 GB
Used: 0.01 GB
Policy:
stripeWidth: 1
spbmProfileGenerationNumber: 0
cacheReservation: 0
spbmProfileName: vSAN Default Storage Policy
forceProvisioning: 0
proportionalCapacity: 0
spbmProfileId: ########-####-####-####-########5bad
hostFailuresToTolerate: 1
CSN: 2
Type: vdisk
Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk (Exists)
Group UUID: ########-####-####-####-########3753
Directory Name: N/A
NOTE: When using "esxcli vsan debug object list" in certain vSAN versions it shows only 100 Objects and it may happen that you don't see the one you are looking for. You will need to add the flag "--all" however, this could impact the host functionality so, it is recommended to run the command and send the output to a txt file. Here it is "esxcli vsan debug object list --all > /tmp/objects.txt". Then you can "less" the file to search for your specific object.
If the vSAN DOM Object UUID is known you can use the following syntax: esxcli vsan debug object list -u <DOM Object UUID>
In the example scenario below, we'll walk through the process of reproducing and correcting this behavior with two example Virtual Machines: "Virtual Machine 1" & "Virtual Machine 2":
"Virtual Machine 1" Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753
"Virtual Machine 2" Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753
1. First, we can obtain the Object UUID of "Virtual Machine 1.vmdk" by simply cat'ing the VMDK Descriptor file -
[root@vSAN-1:~] cat "/vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk"
# Disk DescriptorFile
version=4
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
createType="vmfs"
# Extent description
RW 20971520 VMFS "vsan://########-####-####-####-########3753"
# The Disk Data Base
#DDB
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "1305"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "0046fa2bcb13ce19e6ba05cffffffffe"
ddb.thinProvisioned = "1"
ddb.uuid = "60 00 C2 92 ## ## ## ##-## ## ## ## a2 fe af 7a"
ddb.virtualHWVersion = "14"
2. Next, we can move "Virtual Machine 1.vmdk" to Virtual Machine 2's Directory using a "mv" command:
[root@vSAN-1:~] mv "/vmfs/volumes/vsanDatastore/########-####-####-####-########3753/Virtual Machine 1.vmdk" "/vmfs/volumes/vsanDatastore/########-####-####-####-########3753"
3. Using the vSphere Client UI, we can attach "Virtual Machine 1.vmdk" to Virtual Machine 2:
4. Reviewing the output of esxcli vsan debug object list, we can now see that the VMDK Descriptor is reported as (Missing), even though it still exists:
[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: ########-####-####-####-########3753
Version: 7
Health: healthy
Owner: vSAN-1.gsslabs.org
Size: 10.00 GB
Used: 0.01 GB
Policy:
CSN: 2
stripeWidth: 1
cacheReservation: 0
proportionalCapacity: 0
spbmProfileName: vSAN Default Storage Policy
forceProvisioning: 0
spbmProfileId: ########-####-####-####-########5bad
hostFailuresToTolerate: 1
spbmProfileGenerationNumber: 0
Type: vdisk
Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk (Missing)
Group UUID: ########-####-####-####-########3753
Directory Name: N/A
This is because the Path Attribute for this Object still references the previous directory from Virtual Machine 1. This can be confirmed via an "objtool getAttr" (Object Tool Get Attribute) command:
* Command Syntax: /usr/lib/vmware/osfs/bin/objtool getAttr -u <Object UUID>
[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool getAttr -u ########-####-####-####-########3753
Object Attributes --
UUID:########-####-####-####-########3753
Object type:vsan
Object size:10737418240
User friendly name:(null)
HA metadata:(null)
Allocation type:Thin
Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"########-####-####-####-########5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))
Object class: vdisk
Object capabilities: NONE
Object path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk
Group uuid: ########-####-####-####-########3753
Container uuid: (null)
5. We must update the Path Attribute using an "objtool setAttr" (Object Tool Set Attribute) command.
* Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>
[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool setAttr -u ########-####-####-####-########3753 -d "/vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk"
Object set attribute succeeded
Re-running esxcli vsan debug object list, we can now confirm the VMDK Descriptor File is no longer reported as (Missing):
[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: ########-####-####-####-########3753
Version: 7
Health: healthy
Owner: vSAN-1.gsslabs.org
Size: 10.00 GB
Used: 0.01 GB
Policy:
stripeWidth: 1
spbmProfileGenerationNumber: 0
cacheReservation: 0
spbmProfileName: vSAN Default Storage Policy
forceProvisioning: 0
proportionalCapacity: 0
spbmProfileId: ########-####-####-####-########5bad
hostFailuresToTolerate: 1
CSN: 2
Type: vdisk
Path: /vmfs/volumes/vsan:529e0085########-######dee5e0391b/########-####-####-####-########3753/Virtual Machine 1.vmdk (Exists)
Group UUID: ########-####-####-####-########3753
Directory Name: N/A
NOTE: When using "esxcli vsan debug object list" in certain vSAN versions it shows only 100 Objects and it may happen that you don't see the one you are looking for. You will need to add the flag "--all" however, this could impact the host functionality so, it is recommended to run the command and send the output to a txt file. Here it is "esxcli vsan debug object list --all > /tmp/objects.txt". Then you can "less" the file to search for your specific object.
If the vSAN DOM Object UUID is known you can use the following syntax: esxcli vsan debug object list -u <DOM Object UUID>
Feedback
Yes
No
Powered by
