Rotate monitoring-metric-cert on TKGI clusters

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• Users might see the monitoring-metric-cert show expired or nearing expiry on the Opsman GUI on the Certificates page. 
• This KB will provide steps to replace expiring or expired monitoring-metric-cert certificate on TKGI clusters without replacing all other cluster certs.
• Access to Credhub will be required to replace this certificate individually.

 

Environment

TKGI 1.9+

Resolution

Prerequisite: 

If you have a TKGI cluster where the certificate rotation has failed (tkgi rotate-certificates), you should finish the certificate rotation on this specific TKGI cluster by re-running tkgi rotate-certificates command until it successfully completes.

• Setup credentials for credhub and maestro access (Replace sections in <red> with environment specific variables)
  
  export BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=<secret>
export BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<IP>
export CREDHUB_SERVER="$BOSH_ENVIRONMENT:8844" CREDHUB_CLIENT="$BOSH_CLIENT"
export CREDHUB_SECRET="$BOSH_CLIENT_SECRET" CREDHUB_CA_CERT="$BOSH_CA_CERT"
credhub api https://$BOSH_ENVIRONMENT:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
credhub login

 

List Certificates and Check Current Validity using Maestro:

1. List certificates per cluster deployment (using example Cluster_ID: service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf):
   
   

   maestro --json topology --name /p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert | jq '.topology[].signs[] | "\(.name) \(.versions[].deployment_names)"' | grep monitoring-metric-cert

   "/p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert [\"service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf\"]"
   
   
   

2. Display Valid dates of certificate (using example Cluster_ID: service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf):
   
   

   maestro --json topology --name /p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert | jq '.topology[].signs[] | "\(.name) \(.versions[].valid_until)"' | grep monitoring-metric-cert

   "/p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert 2028-07-24T17:50:25Z"

 

 

Certificate Replacement

1. Gather a list of all impacted clusters with the expired or soon to expire monitoring-metric-cert from Opsman GUI -> Certificates page.
2. Copy the cert name, including path, ie:  /p-bosh/service-instance_<CLUSTER_ID>/monitoring-metric-cert for all clusters for use in later commands
3. From jumpbox or Opsman SSH where the above Prerequisite commands were run:
   
   Replace the leaf cert for EACH of the clusters gathered in step 1 with the following command
   
   # maestro regenerate leaf --name /p-bosh/service-instance_<CLUSTER_ID>/monitoring-metric-cert
   
   
4. Once the maestro command has been run, perform a cluster upgrade for all impacted clusters with the following command:
   
   tkgi upgrade-cluster <CLUSTER_NAME>

 

List Certificates and check Current Validity again using Maestro:

Note: You will now see a second certificate in both commands

1. List certificates per cluster deployment (using example Cluster_ID: service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf):
   
   

   maestro --json topology --name /p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert | jq '.topology[].signs[] | "\(.name) \(.versions[].deployment_names)"' | grep monitoring-metric-cert

   "/p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert [\"service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf\"]"

   "/p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert []"

   
   
   
   

2. Display Valid dates of certificate (using example Cluster_ID: service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf):
   
   

   maestro --json topology --name /p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert | jq '.topology[].signs[] | "\(.name) \(.versions[].valid_until)"' | grep monitoring-metric-cert

   "/p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert 2028-08-06T20:23:28Z"

   "/p-bosh/service-instance_925546b2-fc8a-983n-gok9-3051d99eefcf/monitoring-metric-cert 2028-07-24T17:50:25Z"