Bosh CCK fails to attach Volumes to PKS nodes

Products

• VMware

Issue/Introduction

Symptoms:

·         Running bosh cck against a PKS/TKGI deployment fails with an error message similar to:
Task 34817 | 16:56:05 | Applying problem resolutions: Inconsistent mount information:
Record shows that disk 'disk-74956cc5-816c-4732-8b94-9ea8db8385a8' should be mounted on vm-a002f9d6-b04d-4eab-95bb-107299fb165d.
However it is currently :
        Not mounted in any VM (mount_info_mismatch 83): Reattach disk to instance (00:04:10)
                     L Error: Unknown CPI error 'Unknown' with message 'Failed to add disk 'scsi0:2'.' In 'attach_disk' CPI method (CPI request ID: 'cpi-920928')

·         The output for bosh task <id> --debug shows entries similar to:
{"time":1592499365,"stage":"Applying problem resolutions","tags":[],"total":8,"task":"Inconsistent mount information:\nRecord shows that disk ''disk-74956cc5-816c-4732-8b94-9ea8db8385a8'' should be mounted on vm-a002f9d6-b04d-4eab-95bb-107299fb165d.\nHowever it is currently :\n\tNot mounted in any VM (mount_info_mismatch 83): Reattach disk to instance","index":2,"state":"failed","progress":100,"data":{"error":"Unknown CPI error ''Unknown'' with message ''Failed to add disk ''scsi0:2''.'' in ''attach_disk'' CPI method (CPI request ID: ''cpi-920928'')"}}

Environment

VMware PKS 1.x

Cause

Storage issue preventing access to the file.

Resolution

Check the task and events for the VM from the vSphere client to confirm if the file is missing or locked by another host. 

Another option is to ssh to the ESXi host and check the file for locks and who owns the lock via vmfsfilelockinfo cli. For example "vmfsfilelockinfo -p pcf_disk/disk-74956cc5-816c-4732-8b94-9ea8db8385a8-flat.vmdk"

If the file is locked by another ESXi host. Try ssh to that host and touching the file to see if this releases the lock. For example "touch pcf_disk/disk-74956cc5-816c-4732-8b94-9ea8db8385a8-flat.vmdk". If this does not work a reboot of the ESXi host often will release any locking issues. Otherwise additional troubleshooting the host and storage is required to determine next steps.

Once the storage access issue is resolved, then running bosh cck again to finish re-attaching the missing volumes.