Proactively prevent Kubernetes internal certificates from expiring in vDefend Security Services Platform(SSP)
search cancel

Proactively prevent Kubernetes internal certificates from expiring in vDefend Security Services Platform(SSP)

book

Article ID: 438244

calendar_today

Updated On:

Products

VMware vDefend Firewall with Advanced Threat Prevention VMware vDefend Firewall

Issue/Introduction

After approximately one year of continuous operation of an SSPI/SSP deployment, the management and workload Kubernetes clusters can stop functioning correctly because internal Kubernetes internal certificates (kubeadm control-plane certs and kubelet server certs) expire and are not renewed.

This KB article provides guidance for customers to proactively run the attached script in their SSP and SSPI deployments to help prevent internal certificates from expiring.

Environment

  • vdefend Security Services Platform 5.0.0, 5.1.0, 5.1.1

Cause

Earlier SSPI/SSP versions did not proactively renew the Kubernetes internal certificates. Consequently, after roughly one year of continuous operation, the following certificates expire and cluster operations break:

  • kubeadm control-plane certs on the SSPI host (apiserver, controller-manager, scheduler, etcd peer/server).
  • kubelet server certs on the SSPI host and on workload worker nodes.
  • The on-disk workload kubeconfig can drift out of sync after rotation.

 

Resolution

The script creates two systemd services that run twice a day (at 12:00 AM and 12:00 PM) and check certificate expiry.

If the certificate expiry check results in fewer than 30 days remaining, then the script will immediately trigger a certificate renewal.

If the certificate expiry check results in greater than 30 days, it waits for the clock to hit the 30-day expiry and then auto-renew the certificates, thus keeping the workload kubeconfig and worker-node kubelet certificates in sync.

If you are running SSP/SSPI 5.0, 5.1.0, or 5.1.1, SSH into the SSPI appliance and run:

  1. Download the attached cert-hotfix.tar file from this KB to your local device.

  2. Transfer the downloaded .tar file from your local device to the /tmp folder of the SSPI appliance.

  3. SSH from your local device to the SSPI appliance. (Username should be sysadmin in 5.1.0 and above, and root in SSPI 5.0.0).

  4. Navigate to the /tmp directory.

  5. Untar the file "cert-hotfix.tar" using the command 
    tar xf cert-hotfix.tar
  6. Change directory to cert-hotfix using the command:
    cd cert-hotfix
  7. Run the script install_sspi_remediate_and_sync_kubeconfig.sh using the command:
    sudo ./install_sspi_remediate_and_sync_kubeconfig.sh

Upon completing the above steps, an "INSTALL OK" status will be displayed indicating the installation was successful.

Note:

  • Certificate updates will trigger a rollout of controller nodes, resulting in new controller node names. If firewall rules are configured using node names, they must be updated accordingly.
  • Renewed workload cluster certificates have a validity duration of 1 year.
  • This script must be rerun after SSPI appliance is upgraded to version 5.1.1 or earlier.

Example output of a successful script installation: 

Enabled and started cert-renew.timer and sspi-remediate-sync-kubeconfig.timer (--now so list-timers shows NEXT).

  *+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+*

  *  ^^ S S P I  |  c e r t - r e n e w  +  r e m e d i a t e  ^^  *

  *  ~~ workload kubeconfig  |  I N S T A L L  O K ~~  *

  *+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+**+==+*

  >> * *  A L L  F E A T U R E S  I N  P L A C E  * * <<

  // units+timers | oneshots | --now //

  ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~~

Verification steps:

  • Run the following commands highlighted in bold on the SSPI appliance, to confirm successful script execution. 

Expected output:

root@n3sspi:~/cert-hotfix# 𝘀𝘆𝘀𝘁𝗲𝗺𝗰𝘁𝗹 𝘀𝘁𝗮𝘁𝘂𝘀 𝗰𝗲𝗿𝘁-𝗿𝗲𝗻𝗲𝘄
○ cert-renew.service - Renew control-plane Kubernetes certificates when within 30 days of expiry (kubeadm)
     Loaded: loaded (/etc/systemd/system/cert-renew.service; static)
     Active: inactive (dead) since Thu 2026-04-30 12:00:26 UTC; 7h ago
TriggeredBy: ● cert-renew.timer
    Process: 45227 ExecStart=/etc/init.d/cert-renew (code=exited, status=0/SUCCESS)
   Main PID: 45227 (code=exited, status=0/SUCCESS)
        CPU: 47ms

Apr 30 12:00:26 n3sspi systemd[1]: Starting Renew control-plane Kubernetes certificates when within 30 days of expiry (kubeadm)...
Apr 30 12:00:26 n3sspi root[45258]: cert-renew: Certificates are valid for more than 30 days. No renewal needed.
Apr 30 12:00:26 n3sspi systemd[1]: cert-renew.service: Deactivated successfully.
Apr 30 12:00:26 n3sspi systemd[1]: Finished Renew control-plane Kubernetes certificates when within 30 days of expiry (kubeadm).

 

root@n3sspi:~/cert-hotfix# 𝘀𝘆𝘀𝘁𝗲𝗺𝗰𝘁𝗹 𝘀𝘁𝗮𝘁𝘂𝘀 𝘀𝘀𝗽𝗶-𝗿𝗲𝗺𝗲𝗱𝗶𝗮𝘁𝗲-𝘀𝘆𝗻𝗰-𝗸𝘂𝗯𝗲𝗰𝗼𝗻𝗳𝗶𝗴
○ sspi-remediate-sync-kubeconfig.service - SSPI remediate KCP certificatesExpiryDays and sync workload kubeconfig (clusterctl)
     Loaded: loaded (/etc/systemd/system/sspi-remediate-sync-kubeconfig.service; static)
     Active: inactive (dead) since Thu 2026-04-30 12:00:30 UTC; 7h ago
TriggeredBy: ● sspi-remediate-sync-kubeconfig.timer
    Process: 45228 ExecStart=/usr/local/sbin/sspi_remediate_kcp_and_sync_kubeconfig.sh (code=exited, status=0/SUCCESS)
   Main PID: 45228 (code=exited, status=0/SUCCESS)
        CPU: 1.482s

Apr 30 12:00:28 n3sspi sspi_remediate_kcp_and_sync_kubeconfig.sh[45378]: kubelet-cert: host=n3ssp-md-0-xxxx KUBELET_SERVER_CERT_RENEW_THRESHOLD_DAYS=28 THRESHOLD_SEC=2419200
Apr 30 12:00:29 n3sspi sspi_remediate_kcp_and_sync_kubeconfig.sh[45228]: kubelet-cert: checking worker node n3ssp-md-0-xxxxxx (10.x.x.x)
Apr 30 12:00:29 n3sspi sspi_remediate_kcp_and_sync_kubeconfig.sh[45386]: Warning: Permanently added '10.x.x.x' (ED25519) to the list of known hosts.
Apr 30 12:00:30 n3sspi sspi_remediate_kcp_and_sync_kubeconfig.sh[45386]: kubelet-cert: host=n3ssp-md-0-xxxxx KUBELET_SERVER_CERT_RENEW_THRESHOLD_DAYS=28 THRESHOLD_SEC=2419200
Apr 30 12:00:30 n3sspi sspi_remediate_kcp_and_sync_kubeconfig.sh[45228]: Certificate will not expire
<SKIP>

 

root@n3sspi:~/cert-hotfix# 𝘀𝘆𝘀𝘁𝗲𝗺𝗰𝘁𝗹 𝘀𝘁𝗮𝘁𝘂𝘀 𝘀𝘀𝗽𝗶-𝗿𝗲𝗺𝗲𝗱𝗶𝗮𝘁𝗲-𝘀𝘆𝗻𝗰-𝗸𝘂𝗯𝗲𝗰𝗼𝗻𝗳𝗶𝗴.𝘁𝗶𝗺𝗲𝗿
● sspi-remediate-sync-kubeconfig.timer - Twice-daily timer for SSPI remediate + workload kubeconfig sync (clusterctl)
     Loaded: loaded (/etc/systemd/system/sspi-remediate-sync-kubeconfig.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Tue 2026-04-28 07:16:04 UTC; 2 days ago
    Trigger: Fri 2026-05-01 00:00:00 UTC; 4h 34min left
   Triggers: ● sspi-remediate-sync-kubeconfig.service

Apr 28 07:16:04 n3sspi systemd[1]: Started Twice-daily timer for SSPI remediate + workload kubeconfig sync (clusterctl).

 

root@n3sspi:~/cert-hotfix# 𝘀𝘆𝘀𝘁𝗲𝗺𝗰𝘁𝗹 𝘀𝘁𝗮𝘁𝘂𝘀 𝗰𝗲𝗿𝘁-𝗿𝗲𝗻𝗲𝘄.𝘁𝗶𝗺𝗲𝗿
● cert-renew.timer - Twice-daily timer for control-plane Kubernetes cert check / renewal
    <SKIP>
    Trigger: Fri 2026-05-01 00:00:00 UTC; 4h 33min left
   Triggers: ● cert-renew.service

Apr 28 07:16:04 n3sspi systemd[1]: Started Twice-daily timer for control-plane Kubernetes cert check / renewal.

 

 

Additional Information

This issue will be resolved in a later version of SSP. 

For more details, please contact Broadcom Support by opening a support ticket.

Attachments

cert-hotfix.tar get_app
Powered by Wolken Software