capv user password expiry in Tanzu Kubernetes Grid
search cancel

capv user password expiry in Tanzu Kubernetes Grid

book

Article ID: 319310

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid

Issue/Introduction

TKG 1.6, 2.3.x,2.4.x & 2.5.x which includes TKR 1.28.x,1.27.x,1.26.x 1.23.8, 1.22.11 and 1.21.14, capv user password is set to expire in 60 days in Ubuntu OS and 90 days in Photon OS as part of STIG Hardening.
 

While this is implemented as part of Security Hardening this impacts the ssh login to the nodes once the password has expired.

Environment

Tanzu Kubernetes Grid 1.6.0, 2.3.x, 2.4.x & 2.5.x

Resolution

Changing the capv user password expiry on Existing Clusters:


Method 1 (Recommended)

This would require us to edit the KCP (kubeadmcontrolplane) and kubeadmconfigtemplate objects for each and every cluster from the management cluster context.

Note:

  • This method can be used both before or after the password expiry.
  • Please remember to reserve the IP addresses of these node post recreation as described in our documentation

This would require all the nodes to be rolled out however this is persistent, if a node is recreated changes would persist.

  • Set the context to the Management cluster context 
  • Get the list of kcp objects across all the namespace
kubectl get kcp -A
  • For each and every workload cluster there will be a corresponding KCP object with the name CLUSTER_NAME-control-plane, edit this object using the below command (if the cluster is in a different namespace please make sure to add the namespace using -n flag)
kubectl edit kcp CLUSTER_NAME-control-plane
  • Add the below line to set the password expiry under .spec.kubeadmConfigSpec.preKubeadmCommands 
- chage -m 0 -M -1 capv
  • After the edit the section would look similar to the below snippet:
    preKubeadmCommands:
    - chage -m 0 -M -1 capv
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "127.0.0.1   localhost" >>/etc/hosts
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
    - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
    - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem
      /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'
    useExperimentalRetryJoin: true
  • You can also query this using kubectl command and json query 
 kubectl get kcp CLUSTER_NAME-control-plane -o jsonpath='{.spec.kubeadmConfigSpec.preKubeadmCommands}{"\n"}'

Once the KCP is edited all the control plane nodes would be rolled out one at a time and once the VM is recreated you can validate the changes by logging into the nodes.

Similarly, for the worker nodes, we need to edit the kubeadmconfigtemplate object.

  • Set the context to the Management cluster context 
  • Get the list of kubeadmconfigtemplate objects across all the namespace
kubectl get kubeadmconfigtemplate -A
  • For each and every workload cluster there will be a corresponding kubeadmconfigtemplate object with the name CLUSTER_NAME-md-0 edit this object using the below command
kubectl edit kubeadmconfigtemplate CLUSTER_NAME-md-0 
  • Add the below line to set the password expiry under .spec.template.spec.preKubeadmCommands 
- chage -m 0 -M -1 capv
  • After the edit the section would look similar to the below snippet:
      preKubeadmCommands:
      - chage -m 0 -M -1 capv
      - hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
      - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
      - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
      - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem
        /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'
  • You can also query this using kubectl command and json query 
 kubectl get kubeadmconfigtemplate CLUSTER_NAME-md-0 -o jsonpath='{.spec.template.spec.preKubeadmCommands}{"\n"}'

Unlike KCP which rolls out control plane nodes after successful edit for workload clusters we need to run a patch command on the MachineDeployment object to trigger a rollout.

kubectl patch machinedeployment CLUSTER_NAME-md-0 --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

Once the VM is recreated you can validate the changes by logging into the nodes.

For New Cluster Creation:

If you are newly creating clusters in TKG 1.6 to include this password expiry setting we need to add the below overlay files under ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/

For Control plane: 

Create a file under  ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/ say with the name capv-user-expiry-control-plane.yaml

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")#@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    preKubeadmCommands:
    #! setting the password expiry of capv user to one year
    #@overlay/append
    - chage -m 0 -M -1 capv

For Worker node:

Create a file under  ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/ say with the name capv-user-expiry-worker.yaml

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")#@overlay/match by=overlay.subset({"kind":"KubeadmConfigTemplate"}),expects="1+"
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      preKubeadmCommands:
      #! setting the password expiry of capv user to one year
      #@overlay/append
      - chage -m 0 -M -1 capv


Method 2:

You can use a simple for loop to set the password expiry which is quicker and simple however one thing to note is this is ephemeral (i.e) if a VM is recreated the changes would be lost. This method is only applicable for clusters where the password is not expired yet.

  • Set the context to the cluster 
  • Get the list of nodes in a file using the below command   
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}' > nodes
  • Set the password expiry to never expire, please make sure to change the location of the private key according to your environment
for i in `cat nodes`; do ssh -i /home/ubuntu/.ssh/id_rsa -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -q capv@$i sudo chage -m 0 -M -1 capv; done;
  • You can check if the password expiry is set correctly using the below command
for i in `cat nodes`; do ssh -i /home/ubuntu/.ssh/id_rsa -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -q capv@$i sudo chage -l capv; done;