Update TMC Self Managed Registry Post Deployment
search cancel

Update TMC Self Managed Registry Post Deployment

book

Article ID: 384685

calendar_today

Updated On:

Products

VMware Tanzu Mission Control - SM VMware Tanzu Mission Control Self-Managed

Issue/Introduction

In some situations it is required to change the TMC Self Managed (TMCSM) image registry. This procedure walks through those steps

Environment

Kubernetes workload cluster with a TMCSM deployment configured to pull images from a private registry. 

Note: This process is intrusive and will roll the TMCSM pods and TMCSM service will be disrupted until the pods are back up and running.

  • TMC Self Managed 1.x

Resolution

Prerequisites

  1. Ensure you have a valid backup of the TMCSM deployment. For example a velero backup.
  2. Push the packages and images of TMCSM bundle of the same version as deployed to the new harbor registry.

Note: If the new harbor registry uses different CA certificates, you need to update the CA certificates of the registry for all the clusters including the one where the TMCSM is installed and the workload clusters managed by TMCSM as noted in the TMCSM docs.

 

Procedure

1. Append the new CA certificate in the CR kappcontrollerconfig <cluster-name>-kapp-controller-package:

apiVersion: run.tanzu.vmware.com/v1alpha3
kind: KappControllerConfig
metadata:
  ...
  name: <CLUSTER-NAME>-kapp-controller-package
  namespace: testns
  ...
spec:
  kappController:
    config:
      caCerts: |-
        -----BEGIN CERTIFICATE-----
        …
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        <CONTENT FOR CA CERTIFICATE OF NEW HARBOR/REGISTRY> 
        -----END CERTIFICATE-----
    createNamespace: false
    deployment:
      ...
    globalNamespace: tkg-system
  namespace: tkg-system
status:
  secretRef: <CLUSTER-NAME>-kapp-controller-data-values

2. Add a new CA certificate in the secret <cluster-name>-user-trusted-ca-secret, please note the content of the CA certificate should be double base64 encoded.

apiVersion: v1
data:
  <NEW-HARBOR/REGISTRY-NAME>: <DOUBLE-BASE64-ENCODED-DATA>
kind: Secret
metadata:
  name: <CLUSTER-NAME>-user-trusted-ca-secret
  namespace: tkg-cluster-ns
type: Opaque

4. Update the cluster Custom Resource of the workload cluster to add a new certificate (For VKS/vSphere with Tanzu Clusters or TKGm clusters). If using a non VMware by Broadcom kubernetes platform ensure your cluster nodes trust the new registry CA

spec:
  topology:
    class: tanzukubernetescluster
    ...
    variables:
    - name: trust
      value:
        additionalTrustedCAs:
        - name: <NEW-HARBOR/REGISTRY-CA>
        …

5. Restart the kapp controller that manages the TMCSM deployment

kubectl rollout restart deployment kapp-controller -n tkg-system

6. Update the package repository to the URL of the new Harbor registry to update the TMC SM deployment.

tanzu package repository update tanzu-mission-control-packages --url "<REPOSITORY-URL>/package-repository:<TMCSM-VERSION>" --namespace tmc-local

7. Replace the values of harborProject with the new URL and trustedCAs with the new CA certificate for new Harbor registry in the values.yaml you used to deploy TMCSM.

8. Update the tmc sm package with the new updated values.yaml.

tanzu package installed update tanzu-mission-control -p tmc.tanzu.vmware.com --version "<TMCSM-VERSION>" --values-file /<DIRECTORYLOCATION>/values.yaml --namespace tmc-local

9. Restart the kapp controller that manages the TMCSM deployment again (see step 5)

Check the Results

NOTE: The reconcile of the TMCSM package installs can take up to 25min to go through every TMCSM package install and update each pod image URL.

1. Check if all images of pods of the TMCSM deployment in tmc-local namespace in the workload cluster context, have been updated to the new URL of the registry.

kubectl get pods -n tmc-local -o jsonpath='{range .items[*]}{"\n"}{@.metadata.name} {range @.spec.containers[*]}{.image} {end}}'

2. Check if the value of OLYMPUS_EXTENSIONS_ENDPOINT_FQDN gets updated in configmap olympus-dyncfg in tmc-local namespace.

kubectl get cm -n tmc-local

kubectl get cm -n tmc-local <NAME-OF-OLYMPUS-DYNCFG> -o yaml |grep -i OLYMPUS_EXTENSIONS_ENDPOINT_FQDN

3. (Only for VKS/vSphere with Tanzu or TKGm) Check if the images of TMC agents/extensions running on the supervisor/managment cluster context and workload clusters context that have been attached to TMC, have been updated to the new URL.

kubectl get pods -n vmware-system-tmc -o jsonpath='{range .items[*]}{"\n"}{@.metadata.name} {range @.spec.containers[*]}{.image} {end}}'

Additional Information

Note: There is a known issue found in TMC SM 1.3 and 1.2, were the image registry URL configuration might not be able to get updated for the cluster agents in the managed cluster in the database.

 

The workaround is updating it in the database of the cluster-agent-service manually.

1. Get the PGURL of the database of cluster-agent-service

kubectl get secret cluster-agent-postgres-creds -n tmc-local -oyaml | yq '.data.PGURL|@base64d'

2. Connect to the database in the container of the Postgres

kubectl exec -it $postgres-pod –n tmc-local -- psql “$PGURL”

3. Check if the image registry location is updated or not.

SELECT image_registry FROM extension_definition;

4. If it is still old ones, update the image_registry to new location

UPDATE extension_definition SET image_registry=’<new_registry>’;

5. Wait a few minutes, check again if the tmc agents/extension pods get restarted and updated in the supervisor cluster and the workload clusters 

​​Note: If the supervisor cluster still failed to pull images TMC agents, you may have to reregister the supervisor cluster

tanzu mission-control management-cluster reregister $clusterName -k $KUBECONFIG
Powered by Wolken Software