Reconcile/Recreate tmc-agent-installer on vSphere Supervisor cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

TMC system objects such as tmc-agent-installer resources and/or namespace svc-tmc-cX have been deleted, modified or moved.

While connected to the Supervisor cluster context:

• Check if the svc-tmc-cX namespace is present or was deleted:
  

  kubectl get ns | grep tmc

  
  
  
• If the svc-tmc-cX namespace exists, confirm on the underlying objects:
  

  kubectl get all -n svc-tmc-cX

   

  • See the following for a list of expected healthy resources in a Supervisor cluster that has not yet been registered with Tanzu Mission Control:

    • pod/tmc-agent-installer-<instance>-<id>
    • cronjob.batch/tmc-agent-installer
    • job.batch/tmc-agent-installer-<instance>

 

If any of the above system objects are missing, this can lead to the following scenarios:

• Tanzu Mission Control cannot be installed
  
  
• The Supervisor Cluster cannot be registered with Tanzu Mission Control
  
  
• Failures to register the Supervisor cluster with Tanzu Mission Control
  
  
• The Supervisor Cluster is showing as unhealthy, unknown or disconnected in Tanzu Mission Control
  
  
• Clusters cannot be monitored or managed through Tanzu Mission Control

Environment

vSphere Supervisor

Tanzu Mission Control (TMC)

Tanzu Mission Control Self Managed (TMC-SM)

Cause

The tmc-agent-installer resources and/or namespace have been deleted, modified or moved.

This is only possible while connected to the Supervisor VMs as root which has permissions to edit or delete system objects.

These TMC resources that were deleted do not automatically recreate. It is possible that the manual deletions lead to permanent loss of the system objects.

As a result, TMC operations such as registering the Supervisor cluster with TMC or monitoring the Supervisor cluster through TMC are broken.

The TMC namespace and tmc-agent-installer pod are created when the Supervisor cluster was originally deployed.

Deleting a namespace deletes all objects under that namespace which will break the system and may not be recoverable.

It is not enough to manually "kubectl create ns svc-tmc-cX" the TMC namespace because it will not recreate with the necessary missing system objects.

Resolution

The missing system objects are due to unsupported manual deletions in the Supervisor cluster.

NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permission of a KB or VMware by Broadcom Technical Support.

 

Proper SSO RBAC control should be used to interact with the Supervisor environment. See: About Identity and Access Management for TKG Service Clusters.

Reach out to VMware by Broadcom Technical Support to assess if the missing TMC system objects can be recovered or if the Supervisor cluster must be redeployed.