Reset the expiry of capv user password in Tanzu Kubernetes Grid if ssh is disabled and kubectl dosen’t works in cluster.

Products

• Tanzu Kubernetes Grid

Issue/Introduction

As we know that capv user password is set to expire in 60 days in Ubuntu OS as part of STIG Hardening.While this is implemented as part of Security Hardening this impacts the ssh login to the cluster nodes once the password has expired.

You will get authentication failure error with WARNING: your password has expired.

Environment

Tanzu kubernetes Grid. 

Resolution

However, there is a workaround available.

The chage command in Linux does not change the password itself, but it modifies the password aging and expiration information for a user.

The data affected by chage is stored in the /etc/shadow file.

e.g : 

chage -I -1 -m 0 -M 99999 -E -1

When you use chage to update things like:

• Minimum and maximum days before password change
• Last password change date
• Account expiration date

These fields are updated in /etc/shadow alongside the user’s password hash.

 

Inorder to resolve the deadlock situation to re enable the cluster login we need to perform the below steps :

/etc/shadow: This is where the encrypted user passwords are stored. When you change a password using the passwd command, this file gets updated with the new hashed password and other password-related fields, such as last password change date, password expiration settings, etc.

1. Depending on where you are trying to SSH, such as the the management cluster control plane.

Kindly proceed to shutdown the management control plane VM from the vCenter GUI and remove the disk. 

Note: Please don’t check the option of “Delete files from datastore”. 

To remove the disk post shutting down the VM from the vCenter GUI, follow these steps:

a. Select Action
b. Edit the VM settings.
c. Choose the disk that we need to removed. Make a note of the name.
d. Hover on the right side and click on the (x) to remove.

 

2. Add this disk to a different VM, for example a jumpbox, where you can log in.

On the VM from vCenter GUI, follow these steps: 

a. Select Action.
b. Edit the VM settings.
c. Add a new device.
d. Select the existing hard disk. Browse the VM folder name and add the correct disk.

3. Once the disk is added to the jumpbox VM  follow these steps.

a. To mount this disk to the jumpbox vm, first run “sudo fdisk -l”.

Partition table entries are not in disk order.

Disk /dev/sdb: 40 GiB, 42949672960 bytes, 83886080 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0C603E73-508C-4481-8514-8C893DD57697

Device  Start   End Sectors Size Type
/dev/sdb1 2048  10239  8192 4M BIOS boot
/dev/sdb2 10240  30719  20480 10M EFI System
/dev/sdb3 30720 83886046 83855327 40G Linux filesystem

b. Create a directory to mount “/dev/sdb<>”  using “mkdir master-node”.

c. Run the mount command:

sudo mount /dev/sdb3 /home/ubuntu/master-node/

d. cd to “master-node/etc” you will find a file in the directory name as shadow. 

Take a backup of the shadow file once before edit. 

cp shadow shadow_backup

e. Edit the shadow file carefully using vi editor.

f. You will find the user capv present in that file.    

Modifies the password aging and expiration settings for the user capv. Here’s a breakdown of the options used:

• -I -1: This sets the inactive password period to -1, meaning the account will never become inactive (i.e., it won’t be disabled even after the password expires).
• -m 0: This sets the minimum number of days between password changes to 0, meaning the user can change their password at any time.
• -M 99999: This sets the maximum number of days that a password remains valid to 99999, effectively making the password never expire (since 99999 days is over 273 years).
• -E -1: This sets the account expiration date to -1, meaning the account will never expire.
• capv: This is the username whose settings are being modified.

So your capv user setting should looks something similar to below: 

Example of /etc/shadow file entry:

capv:!:19000:0:99999:7:::

• capv: Username
• ! : Encrypted password
• 19000: Last password change date (in days since Jan 1, 1970)
• 0: Minimum number of days before password change
• 99999: Maximum number of days the password is valid
• 7: Number of days of warning before password expiration
• Empty 7th and 8th fields: Inactive period and account expiration date (when set with -1, these remain empty or are set accordingly).

g. Next unmount the disk from the jumpbox:

sudo umount /master-node

4. The last step is to remove the disk from the jumpbox and re-add it to the management control plane VM in the vCenter GUI.

On the VM from vCenter GUI, follow these steps:

a. On the master control plane VM Select Action.

b. Add the existing Hard disk. Browse the VM folder name and add the correct disk.