How to automatically detect orphaned service instances which lack a corresponding bosh deployment

Products

• VMware Tanzu Application Service

Issue/Introduction

Sometimes services instances which have been created in CAPI via cf CLI or Apps Manager may not have a bosh deployment present. The bosh deployment, when it exists, has a name like "service_instance-<guid>" where <guid> is the GUID of the service. 

Reasons for the bosh deployment being absent could include:

1. Infrastructure failure or poor performance prevented the operation from completing in a timely fashion
2. A limit on the maximum number of on-demand instances hindered the creation of the deployment
3. The deployment was created but was subsequently deleted.

Finding these "orphaned" service instance can be manually intensive and time-consuming.

Environment

This script can be run on any Linux VM that has meets the following requirements:

1. It has network access to the Tanzu foundation which you want to inspect
2. cf CLI (version 7 or higher) is installed and logged in with appropriate credentials
3. bosh command line installed and logged in with appropriate credentials

Resolution

The following script can be customized to report on service types of interest to you. To do so, modify line 8, the array of values for the variable "service_types".

#!/bin/bash

# Run as ./get_orphan_si.sh > output.txt

set -x

# Define your list of service types
service_types=("postgres" "p.redis" "p.mysql" "p.rabbitmq")  #<<<<<<  FILTER

printpage()
{
    {
        while read -r siguid spaceurl siname service_url; do

            echo $dplist | grep "service-instance_$siguid" > /dev/null
            si_exist=$?
            if [ $si_exist -ne 0 ]; then
                read -r sispacename orgurl < <(cf curl $spaceurl | jq -r '"\(.entity.name)  \(.entity.organization_url)"' && printf '\0')
                siorgname=$(cf curl $orgurl | jq -r .entity.name)
                read -r service_name broker_name < <(cf curl $service_url | jq -r '"\(.entity.label)  \(.entity.service_broker_name)"' && printf '\0')

                # Check if the service_name is in the list of service_types
                if [[ " ${service_types[*]} " == *" $service_name "* ]]; then
                    printf "%-32s %-48s %-32s %-32s %-32s %-76s\n" "$siorgname" "$sispacename" "$broker_name" "$siname" "$service_name" "$siguid"
                fi
            fi
        done
    } < <(echo $resources | jq -r '"\(.metadata.guid)  \(.entity.space_url)  \(.entity.name) \(.entity.service_url)"' && printf '\0')
}

## main
# check if cf/jq exist
which cf > /dev/null 2>&1 ; existing=$?;
if [ $existing -ne 0 ]; then
    echo "cf CLI not available. Please install it or set PATH env variable properly!"
    exit 1
fi

which jq > /dev/null 2>&1 ; existing=$?;
if [ $existing -ne 0 ]; then
    echo "jq utility not available. Please install it or set PATH env variable properly!"
    exit 1
fi

# check if cf logged in
cf orgs > /dev/null 2>&1 ; login_status=$?;
if [ $login_status -ne 0 ]; then
    echo "Not logged in yet. Please do 'cf login' first!"
    exit 1
fi

# check bosh env setup
bosh tasks  --recent=1 &> /dev/null
if [ $? -gt 0 ]
then
    echo "bosh environment variables need to be set.  bosh command line options cant be retrived from https://opsman/api/v0/deployed/director/credentials/bosh_commandline_credentials"
    exit 1
fi

# Get list of deployment names
dplist=$(bosh curl /deployments | jq '.[].name' | tr '"' ' ' | tr -d '\n')

## get first page
onepage=$(cf curl /v2/service_instances)
next_url=$(echo $onepage | jq -r '.next_url')
resources=$(echo $onepage | jq -r '.resources[]')

printf "Checking orphan service instances without corresponding BOSH deployments ...\n\n"
printf "%-32s %-48s %-32s %-32s %-32s %-76s\n" "ORG" "SPACE" "BROKER" "SI_NAME" "SERVICE" "SI_GUID"

echo $(yes - | head -n122)
printpage

## get remaining pages
while [ $next_url != "null" ]; do
    onepage=$(cf curl $next_url)
    echo $onepage >> /tmp/si.out
    next_url=$(echo $onepage | jq -r '.next_url')
    resources=$(echo $onepage | jq -r '.resources[]')
    printpage
done

 

The output should look like this:

hecking orphan service instances without correspoding BOSH deployments ...

ORG                              SPACE                                            BROKER                           SI_NAME                          SERVICE                          SI_GUID                                                                     
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Sandbox                          Development                                      postgres-odb                     bildr-db-test                    postgres                         xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

 

Using the script output, you can use the "org", "space", and "si_name" (service instance name) and the "cf purge-service-instance" command to get rid of the orphaned SI.