Pivotal Cloud Foundry "On Demand" Services are not recreated when "Recreate all VMs option" Option is Selected in Director Tile

Products

• Operations Manager

Issue/Introduction

Symptoms:

PCF Operations (Ops) Manager allows operators to trigger a recreate of all VMs via the “Recreate all VMs” Director tile configuration option. This does not recreate the On Demand service instances deployed by tiles dynamically.

Impact

User on-demand service instances will not be recreated when the operator expects them to be. For example, when following the vSphere migrating datastore process, the on-demand instances VMs will not be recreated.

Environment

Resolution

After successfully applying changes in PCF OpsManager, you need to manually trigger the recreate of the on-demand deployment. Please note that not all Bosh command examples below assume that the user is ssh'ed into Operation Manager VM.

BOSH V2 CLI (Note Operations Manager renames bosh_cli to bosh2 for version 2)

1. Identify the tiles that are on-demand enabled in your PCF OpsManager environment. These tiles will require a service network
2. Login to BOSH using the admin credentials found in the Director tile

• Alternatively, you can use the BOSH team admin for the tile. This can be found on the credentials tab of the Director Tile.

BOSH V1 CLI

• This command will show all of the deployments and their given BOSH team. The team relates back to the tile that created the deployment. See BOSH deployment management documentation for more details

  bosh2 -e my-env deployments

• Identify a list of all of the on-demand service instances. The name will be in the format "service-instance_$guid." This GUID maps back to the Cloud Foundry service instance GUID
• For each of the On Demand deployment, run a bosh recreate. See BOSH deployment management documentation for more details

  bosh2 -e my-env -d my-dep recreate 

  1. Identify the tiles that are on-demand enabled in your PCF OpsManager environment. These tiles will require a service network
  2. Login to BOSH using the admin credentials found in the Director tile. Bosh CLI is pre-installed on the Operations Manager VM

     bosh -n login admin password

  3. Get a list of deployments using the following command. We will need the deployment name of the On Demand Service instances. The deployment name will be named like "service-instance_$guid" as per the below example. Where $guid maps back to the Cloud Foundry service instance GUID

     $ bosh deployments
     service-instance_052c4e15-e295-4658-999e-efd1a9e9efd7  mysql/0.6.0               
     bosh-warden-boshlite-ubuntu-trusty-go_agent/3363.9 latest
     

  4. Identify a list of all of the on-demand service instance deployments
  5. For each of the deployment, run the following steps
     1. Download the deployment manifest

        bosh download manifest service-instance_$guid service-instance_$guid.yml

     2. Update the manifests "director_uuid" param with the value from the Bosh status command

        bosh status --uuid

     3. Select the deployment

        bosh deployment service-instance_$guid.yml

     4. Recreate with this Bosh command to force all the VMs in the service instance deployment to get recreated

        bosh recreate