How to Restore an Accidentally Deleted Tile or a BOSH or Ops Manager Deployment

Products

Operations Manager

Issue/Introduction

While Ops Manager & Bosh offer some safeguards to prevent an operator from mistakenly deleting a tile or deployment that is being used, accidents do happen. Fortunately, in some cases, it is possible to recover from an accidentally deleted tile or Bosh deployment.

With some manual effort, it is possible to recover from accidentally deleting a deploy so long as Bosh still has the persistent disks in it's orphaned disk list.

This article walks through the process of recovering.

Environment

Resolution

Procedure to restore an accidentally deleted deployment. We’re considering RabbitMQ as an example deployment to explain the restore process.

Step 1

`bosh tasks recent --no-filter` will list the recent tasks. By default, Bosh will only store 100 recent tasks in its database.

Make sure you see the delete deployment task as below

Step 2

Execute `bosh disks --orphaned` and see the persistent disks for the deleted deployment are still listed as orphaned disks

Note: If you do not see any orphaned disks then the persistent data has been deleted, and it's not possible to restore the deleted deployment. In this case, this KB is not applicable to your use case, so you would need to reinstall the missing tile and restore your data from a backup.

Now identify the right VM’s to which these disks were attached; follow the below process.

Step 3

Execute `bosh task <task-number>`, where `<task-number>` is the task number from step #1. In our example, it’s `bosh task 9291`, which will list the deleted VM id's like below

Step 4

Comparing the GUID from the **Instance Name** from `Step 2` output and `Step 3,` make a list of the corresponding VM and disk mappings.

In this scenario, the following is the VM-Disk mapping

VM Disk ID

rabbitmq-server/0 -- disk-abb98ec5-c0ee-4f85-bd6d-ac2d23275396

rabbitmq-server/1 -- disk-fd5e3cb4-012e-42f8-a726-f4b73b1449e3

rabbitmq-server/2 -- disk-33e8a197-aaee-45ad-854f-5f40aeefee4a

Step 5

If the deployment is deleted via ops manager, add the deleted tile back to Ops Manager and click on apply changes. In this example, adding RabbitMq back and Apply Changes will create a new deployment.

Or

For a Bosh deleted deployment and if tile still appears in Ops Manager web page then just click on Apply Changes from Ops Manager.

Once the deployment is successfully completed, the deleted tile or deployment should be available again. However, it won’t contain any data. In our example, the queues section from the new deployment would display nothing (screenshot), because there is no user data. Other tiles will vary, but should also appear as if there is no data in them.

Step 6

Stop the VMs in the new deployment. For our RabbitMQ example, that would be:

`bosh stop rabbitmq-broker`

`bosh stop rabbitmq-haproxy`

`bosh stop rabbitmq-server`

If you need help with getting started with bosh, please refer this document here: https://docs.pivotal.io/pivotalcf/1-11/customizing/trouble-advanced.html#prepare

Step 7

Start attaching the disks for the respective VM’s as below

`bosh attach disk <job_name> <id> <disk_cid>`

For the example scenario, below are the 3 disk attach commands for 3 server VM’s

bosh attach disk rabbitmq-server 5fa7b46d-5348-43f1-8acd-8046b3515c33 disk-abb98ec5-c0ee-4f85-bd6d-ac2d23275396
bosh attach disk rabbitmq-server c2a040c2-994a-4fb1-8237-45adad3035d8 disk-fd5e3cb4-012e-42f8-a726-f4b73b1449e3
bosh attach disk rabbitmq-server 8e2da826-de47-4150-9e19-404427ef6b3a disk-33e8a197-aaee-45ad-854f-5f40aeefee4a

Step 8

Start the VMs in your new deployment. For our RabbitMQ example, start the jobs as per the below sequence:

`bosh start rabbitmq-server`

`bosh start rabbitmq-broker`

`bosh start rabbitmq-haproxy`

You should now see all the data as the original. The example scenario all the queues are active including unconsumed messages in the queues and the application, which is being used for testing so we can confirm that apps were able to consume the messages from its queue.