Recreation of a vm by Bosh fails with disk not found error in Tanzu Kubernetes Grid Integrated

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Application Service

Issue/Introduction

You will see the below CPI errors upon checking the bosh task output.

Not mounted in any VM (mount_info_mismatch 59): Reattach disk and reboot instance (00:02:03)

                       L Error: Unknown CPI error 'Unknown' with message 'Failed to add disk scsi0:3.' in 'attach_disk' CPI method (CPI request ID: 'cpi-675887')

Task 112446 | 01:14:53 | Error: Error resolving problem '1551': Unknown CPI error 'Unknown' with message 'Failed to add disk scsi0:3.' in 'attach_disk' CPI method (CPI request ID: 'cpi-316692')

Error resolving problem '1548': Agent is responding now, skipping resolution

Error resolving problem '1547': Agent is responding now, skipping resolution

Error resolving problem '1549': Agent is responding now, skipping resolution

Error resolving problem '1546': CPI error 'Bosh::Clouds::DiskNotFound' with message 'Could not find disk with id 'disk-xxxxxxxxx-013c-40da-b501-3ebb45c7fb25'' in 'create_vm' CPI method (CPI request ID: 'cpi-149577')

Error resolving problem '1554': Unknown CPI error 'Unknown' with message 'Failed to add disk scsi0:3.' in 'attach_disk' CPI method (CPI request ID: 'cpi-582919')

Error resolving problem '1552': Unknown CPI error 'Unknown' with message 'Failed to add disk scsi0:3.' in 'attach_disk' CPI method (CPI request ID: 'cpi-791774')

Error resolving problem '1550': Unknown CPI error 'Unknown' with message 'Failed to add disk scsi0:3.' in 'attach_disk' CPI method (CPI request ID: 'cpi-166004')

Error resolving problem '1553': Unknown CPI error 'Unknown' with message 'Failed to add disk scsi0:3.' in 'attach_disk' CPI method (CPI request ID: 'cpi-675887')

 

 

Cause

If the persistent disk is deleted from the underlying infrastructure Bosh will try to locate it and fail with the above error.

Disk can be missing if its deleted by human error or due to storage vmotion

Resolution

Since the disk is missing in the underlying datastore, data will not be persistent. However to bring up the vm you can recreate using the below options which will recreate the vm and also Bosh will create a new persistent disk and attach it to the newly created vm.

Please do not perform this step unless you are sure the old disk is no longer available. 

And do not perform this step on a Master node unless you have a copy of ETCD data.

If you are unsure about running this command, please contact Tanzu Support for assistance. 

bosh -d recreate worker/uuid --no-converge --skip-drain --fix