BOSH Director persistent disk re-attachment issues on reboot

Products

• Operations Manager

Issue/Introduction

Symptoms:

• BOSH Director's persistent disk was recently resized
• The BOSH Director was restarted while multiple disks were attached.  This can happen during a persistent disk resize as BOSH will copy data from the old disk to the new disk.
• BOSH Director has become unresponsive
• You can’t authenticate to or access BOSH Director with the BOSH CLI
• Rebooting the BOSH Director does not resolve the issue

Environment

Cause

Change of persistent disk size where multiple disks are attached and BOSH Director VM gets rebooted, will cause the BOSH Agent to not properly restart and put the BOSH Director VM into a hung state due to improper remount of the persistent disk.

Resolution

• As of this writing, there is a Tracker story to address this known issue.
• As a workaround, trigger a director VM recreation via non-persistent-disk change. e.g. change the ephemeral disk size.  A non-persistent-disk change should remove the multiple (and stale) disk references and place the BOSH Director back to a normal state.