VM disk errors for vCloud Availability placeholder independent disks repeat in the events view of the vSphere Client

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• In the Events view of the vSphere Client, you see entries similar to:

"Cannot complete the operation because the file or folder [XX-AAD2-BB20-F-01] C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26)/C4-b82b42bb-####-####-####-########ac2
(2f83fada-####-####-####-########b26).vmdk already exists"

"Cannot complete the operation because the file or folder [XX-AAD2-BB20-F-01] C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26)/C4-b82b42bb-####-####-####-########ac2
(2f83fada-####-####-####-########b26)_changed.vmdk already exists"

• In the vCloud Availability Portal, all replications are in a healthy state.
• In /var/log/vmware/vpxd/vpxd.log on the vCenter Server appliance, you see entries similar to:

2019-12-17T17:09:15.551Z info vpxd[7F0CDECD9700] [Originator@6876 sub=vpxLro opID=34d9b0b5-####-####-####-########5ee_ND_GI-cd] [VpxLRO] -- BEGIN task-333656 -- virtualDiskManager -- vim.VirtualDiskManager.createVirtualDisk -- 527c5a88-####-####-####-########612(521c9e62-####-####-####-########e53)
2019-12-17T17:09:15.585Z info vpxd[7F0D05A75700] [Originator@6876 sub=vpxLro opID=34d9b0b5-####-####-####-########5ee_ND_GI-cd] [VpxLRO] -- FINISH task-333656
2019-12-17T17:09:15.585Z info vpxd[7F0D05A75700] [Originator@6876 sub=Default opID=34d9b0b5-####-####-####-########5ee_ND_GI-cd] [VpxLRO] -- ERROR task-333656 -- virtualDiskManager -- vim.VirtualDiskManager.createVirtualDisk: vim.fault.FileAlreadyExists:
--> Result:
--> (vim.fault.FileAlreadyExists) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    file = "[XX-AAD2-BB20-F-01] C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26)/C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26).vmdk"
-->    msg = "Cannot complete the operation because the file or folder [XX-AAD2-BB20-F-01] C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26)/C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26).vmdk already exists"
--> }
--> Args:
-->
--> Arg name:
--> "[XX-AAD2-BB20-F-01] C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26)/C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26).vmdk"
--> Arg datacenter:
--> 'vim.Datacenter:datacenter-21'
--> Arg spec:
--> (vim.VirtualDiskManager.FileBackedVirtualDiskSpec) {
-->    diskType = "thin",
-->    adapterType = "lsiLogic",
-->    capacityKb = 41918464,
-->    profile = <unset>,
-->    crypto = (vim.encryption.CryptoSpec) null

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCloud Availability 3.5.x

Cause

This issue can occur due to various environmental issues external to vCloud Availability at the time the placeholder independent disk size is being modified.

Common environmental issues that can cause this are:

• vCenter Server performance resulting in slow response times.
• Network issues impacting component communication.

Resolution

This issue is resolved in vCloud Availability 3.5.1, available at Support Documents and Downloads (broadcom.com).

Workaround:
To work around this issue if you do not want to upgrade, delete the placeholder independent disk VMDK files that are referenced in the vCenter Server events. On the next independent disk size check, the VMDK files will be recreated.

Note: Active replications will not be impacted by following the steps below, only vCloud Director storage reporting will be impacted until the new VDMK files are recreated.

1. In the vSphere Client go to Monitor > Events for the affected vCenter Server.
2. Identify the unique C4 VMDK files reported in the events.

Example:
"Cannot complete the operation because the file or folder [DATASTORE] C4-b82b42bb-####-####-####-########ac2 (2f83fada-####-####-####-########b26)/C4-b82b42bb-####-####-####-########ac2
(2f83fada-####-####-####-########b26)_changed.vmdk already exists"

3. SSH to an ESXi host that has access to the datastore the C4 VM resides on and log in as root.
4. Navigate to the C4 VM directory obtained from the event message.

Example:
cd /vmfs/volumes/DATASTORE/"C4-b82b42bb-5912-40e3-b595-eb5351877ac2 (2f83fada-1c7a-4110-9b30-ac88a08c2b26)"

5. List the contents of the directory.

Example:
-rw-r--r--    1 root     root           0 Oct  9 21:36 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26).vmsd
-rw-r--r--    1 root     root         303 Oct  9 21:36 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26)-2bc9383d.hlog
-rw-------    1 root     root      1.6G Nov 29 22:23 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26)-flat.vmdk
-rwxr-xr-x    1 root     root        2.1K Nov 29 22:23 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26).vmx
-rw-------    1 root     root         546 Nov 29 22:23 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26).vmdk
-rw-------    1 root     root      1.6G Nov 29 23:03 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26)_changed-flat.vmdk
-rw-------    1 root     root         554 Nov 29 23:04 C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26)_changed.vmdk

6. Remove both the -flat.vmk and .vmdk files for the disk referenced in the event message.

Example:
rm C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26))_changed.vmdk
rm C4-b82b42bb-####-####-####-########ac2(2f83fada-####-####-####-########b26))_changed-flat.vmdk

7. Repeat steps 4-6 for all other VMDK files referenced in vCenter events.