BOSH attaching persistent disk failed with 'Provided backing file is not accessible from host'

Products

• VMware Tanzu Application Service

Issue/Introduction

When BOSH tried to attach existing persistent disk to a VM on vSphere,  it failed with CPI error 'Invalid configuration for device '0'.', as shown below.

Updating deployment:
  Expected task '10122876' to succeed but state is 'error'
Task 10122876 | 17:45:52 | L executing post-stop: spring-cloud-services/55e77801-####-8e659adc383d (0) (canary) (00:03:31)
                         L Error: Unknown CPI error 'Unknown' with message 'Invalid configuration for device '0'.' in 'attach_disk' CPI method (CPI request ID: 'cpi-741273')
Task 10122876 | 17:48:59 | Error: Unknown CPI error 'Unknown' with message 'Invalid configuration for device '0'.' in 'attach_disk' CPI method (CPI request ID: 'cpi-741273')

BOSH task debug logs contained more fault message 'Provided backing file is not accessible from host.,Device: VirtualDisk.'

D, [2024-10-14T17:47:15.714934 #1348195] DEBUG -- [req_id cpi-741273]: disk disk-058a6f00-####-97f3f3df2dad found in: (VSphereCloud::Resources::Datastore (name="####-##", mob="<[Vim.Datastore] ####-##>"))
......
I, [2024-10-14T17:47:15.985815 #1348195]  INFO -- [req_id cpi-741273]: Attaching disk
D, [2024-10-14T17:47:15.985903 #1348195] DEBUG -- [req_id cpi-741273]: Running method 'ReconfigVM_Task'...
D, [2024-10-14T17:47:15.994947 #1348195] DEBUG -- [req_id cpi-741273]: Starting task 'VirtualMachine.reconfigure'...
W, [2024-10-14T17:47:17.004463 #1348195]  WARN -- [req_id cpi-741273]: Error running task 'VirtualMachine.reconfigure'. Failed with message 'Invalid configuration for device '0'.' and fault message 'Provided backing file is not accessible from host.,Device: VirtualDisk.'.
......

The message 'Invalid configuration for device '0'.' and fault message 'Provided backing file is not accessible from host.,Device: VirtualDisk.' were also visible on vSphere client for the related VM. Therefore, the errors were actually produced by vSphere, not TAS/BOSH. 

Cause

Error message 'Provided backing file is not accessible from host.' means it's unable to access the persistent disk file from the ESXi host where the related VM was deployed. vSphere team should be engaged to do further investigation from infrastructure perspective. It might be a common issue to all ESXi hosts for accessing the specific datastore or only a few ESXi hosts have such problem.

Resolution

In this case there were 2 ESXi hosts in the cluster having problem with accessing the datastore where the persistent disk file resides. vSphere team solved the issue with the following steps.

1) Put the problematic ESXi host into maintenance mode first

2) Reboot the ESXi host and pull it out of maintenance mode

3) Run above steps for all problematic hosts

4) After the ESXi hosts are reboot and join the cluster, kick off "Apply Changes" from Ops Manager to start a new BOSH deploy