The metrics_server process is in a "does not exist" status on the bosh director VM

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

Symptoms:

• In the Opsman UI, the Bosh director tile status has a warning 
• The metrics_server process is in a "does not exist" status on the bosh director VM
• Manually starting the process does note affect the status
• All bosh operations continue to run without issue  
• You see messages similar to the following in the metrics_server.stderr.log on the bosh director VM:

from /var/vcap/packages/director/bin/bosh-director-metrics-server:29:in `<main>'
/var/vcap/data/packages/director/4881f3db8885fcd9c9ecb7ace4b500377f6e9362/gem_home/ruby/2.6.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/cloud_manifest_parser.rb:120:in `parse_vm_extensions': Duplicate vm extension name 'disk_enable_uuid' (Bosh::Director::DeploymentDuplicateVmExtensionName)

• The bosh configuration has a duplicate vm extension named 'disk_enable_uuid'. Note: You can validate by running the bosh configs command and looking for output similar to the following:

  bosh config 6
  
  Using environment 'xxxxxxxxxx.xxxx.xxxxxxxx.xxxx.xxxxxxxxx' as client 'ops_manager'
  
  ID          6
  Type        cloud
  Name        pivotal-container-service-a302e8c394749f450fc1
  Created At  2020-05-11 07:11:40 UTC
  Content     vm_extensions:
              - cloud_properties:
                  vmx_options:
                    disk.enableUUID: "1"
                name: disk_enable_uuid
              - cloud_properties:
                  upgrade_hw_version: true
                name: set_version_hardware
  
  bosh config 17
  
  Using environment 'xxxxxxxxxx.xxxx.xxxxxxxx.xxxx.xxxxxxxxx' as client 'ops_manager'
  
  ID          17
  Type        cloud
  Name        pivotal-container-service-8733db4721bf2071cdc2
  Created At  2020-05-12 09:33:59 UTC
  Content     vm_extensions:
              - cloud_properties:
                  vmx_options:
                    disk.enableUUID: "1"
                name: disk_enable_uuid
              - cloud_properties:
                  upgrade_hw_version: true
                name: set_version_hardware

  Note: From the bosh configs output, you can see that these are both for the PKS VM.

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Resolution

The duplicate configs have names ending in a GUID which should correspond to a bosh deployment for TKGi/PKS. To determine which config is valid and which can be deleted, list the deployment from bosh:

bosh deployments --column=name

The valid bosh config will have a GUID which matches the GUID for the pivotal-container-service-GUID.

After identifying the duplicate bosh config, delete it by running bosh delete-config <ID>.