Cannot complete the operation because the file or folder already exists - Invalid datastore path

Products

• VMware Tanzu Application Service

Issue/Introduction

The apply change fails during the detach disk phase of recreating the VM. VMs are failing when detaching because there is a datastore path mismatch between BOSH and vSphere. 

When Bosh goes to look up the disk that it's detaching, it can't find it in the datastore that does not exist. When Bosh searches for the disk in vSAN, it gets a path back that isn't in the VMs folder, where it normally creates a persistent disk (due to vSAN symlinking). To recover from this, Bosh tries to move the disk into the folder where it's expected, but the public name (symlink) is already there, so the move operation fails with the already exists failure below. This causes the detach disk process to exit with failure. 

Task 7056666 | 20:43:45 | Updating instance backup_restore: backup_restore/#####  (0) (canary) (00:02:05)
                        L Error: Unknown CPI error 'Unknown' with message 'Cannot complete the operation because the file or folder [<New-Folder-Name>] ####-bosh-disks/disk-####.vmdk already exists' in 'detach_disk' CPI method (CPI request ID: 'cpi-925541')
Task 7056666 | 20:43:49 | Updating instance mysql_proxy: mysql_proxy/#####  (0) (canary)
Task 7056666 | 20:43:49 | L configuring jobs: mysql_proxy/##### (0) (canary)
Task 7056666 | 20:43:49 | L executing pre-start: mysql_proxy/#####  (0) (canary)
Task 7056666 | 20:43:50 | L starting jobs: mysql_proxy/#####  (0) (canary)
Task 7056666 | 20:44:00 | Updating instance mysql: mysql/#####  (0) (canary) (00:02:20)
                        L Error: Unknown CPI error 'Unknown' with message 'Cannot complete the operation because the file or folder [<New-Folder-Name>] ####-bosh-disks/disk-####.vmdk already exists' in 'detach_disk' CPI method (CPI request ID: 'cpi-623025')

In the task --cpi logs are the following error:

Starting detach_disk...
Detaching disk: disk-#### from vm: backup_restore_cf-...
Found 1 persistent disk(s)...
Detaching: [<New-Folder-Name>] 61876e5d-ea5b-b8f8-d0cd-e4434b4b6810/disk-####.vmdk
Current file path is 61876e5d-...
Original file path is ####-bosh-disks/disk-####-...
Trying to find disk in: [<Old-Folder-Name>-...]
WARN -- Error running task 'host.DatastoreBrowser.search'. Failed with message 'Invalid datastore path'

We can confirm and see this value set in vApp Options (Search for VM -> Configure -> vApp Options -> Properties Value) in vSphere.

Cause

When performing datastore name changes, the VMs must be recreated for the datastore names to take effect. If they are not, it will result in a mismatch.

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-operations-manager/3-0/tanzu-ops-manager/refarch-vsphere-vsphere_migrate_datastore.html

Resolution

Please reach out to Tanzu Support and open a case to assist with resolving this issue.