Rotating VMSP TLS certificates
search cancel

Rotating VMSP TLS certificates

book

Article ID: 392017

calendar_today

Updated On: 06-17-2025

Products

VCF Automation

Issue/Introduction

In the event of a security compromise and the customer needs to replace all SSL certificates, the customer must connect to the Kubernetes cluster and manually perform a rotation of the certificates used internally to the cluster.

Environment

VCF Automation 9.0

Cause

VMSP system TLS certificates are compromised

Resolution

1. Obtain the Kubeconfig from one of the following 3 options

  • SSH to one of the nodes and use /etc/kubernetes/admin.conf
  • Connect to LCM and get it from the locker
  • Execute /var/lib/vrlcm/fetch-kubeconfig.py <lcm host> <admin_pass> <linux_root_pass> <env_id> - the environment ID is listed in the URL of the application in the LCM window


2. Delete VCF cluster CA backup secrets

export KUBECONFIG=path_to_kubeconfig_file
kubectl -n vmsp-platform get secret -o name| grep -i "vcf-cluster-ca-copy" | xargs -I {} kubectl -n vmsp-platform delete {}

3. Delete platform CA, vcf-cluster-ca-secret secret

export KUBECONFIG=path_to_kubeconfig_file
kubectl -n vmsp-platform delete secret vcf-cluster-ca-secret

4. Reissuing of all certificates

#!/bin/bash
export KUBECONFIG=path_to_kubeconfig_file
echo "Patching all certificates"
for namespace in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
   echo "Patching certificates in namespace: ${namespace}"
   kubectl -n ${namespace} get certificates -o name | xargs -I {} kubectl -n ${namespace} patch {} --type=json --subresource status --patch '[{ "op": "add","path": "/status/conditions/-","value": {"type":"Issuing", "status":"True", "lastTransitionTime":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}}]'
done
echo "Finished patching certificates in all namespaces."

Watch Kubernetes Certificate Requests to ensure that all certificates were re-generated.

You can monitor the progress with the following command

export KUBECONFIG=path_to_kubeconfig_file
kubectl get certificaterequest -A --watch --request-timeout='2m'

5. Restarting the Platform

To ensure that all applications use the newly generated certificates we need to restart the platform.

This could be done via LCM UI by triggering Power OFF, Power ON functionality.