Error: vmware-system-user account expired

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

When using SSH to connect to a node via the vmware-system-user account, it has expired

Environment

1.23.x

1.24.x

Cause

The VMware-system-user password is set to expire in 60 days as part of STIG Hardening.

If the Guest Cluster's API server is unresponsive, the pass_expiry will not be applied.yaml noted in the resolution, which will prevent the correction of the VMware-system-user account. This presents a condition where the API server cannot be corrected because the SSH account is locked, and the account can't be corrected because the API server is down. 

To correct this condition, it is required to reboot the Guest Cluster nodes and boot into the grub menu via the VM Console, which will allow the user to clear expiration on the vmware-system-user account. 

Resolution

1. Navigate to the workload cluster's VMs in the vSphere web client.
2. Click on: ACTIONS -> Edit Settings on the first workload cluster VM, or right-click on the VM in the Inventory view for Edit Settings.
3. Go to "VM Options" -> expand "Boot Options" and set the "Boot Delay" to 5000 milliseconds. This will make it easier to enter GNU GRUB at the next VM start.
4. Open the VM console to the workload cluster VM. These docs provide details on the following process:
   Resetting the root password on a Photon appliance in VMware Aria Automation 8.x
   
   
5. Restart the workload cluster VM from the vSphere Web Client.
6. Return to the VM console. When the Photon boot screen starts, interrupt the boot by pressing 'e'.
7. At the end of the Linux line, add "rw init=/bin/bash", then press F10 to boot:
   
   
   
   
   
   
   
8. If using Linux or Ubuntu based VM's, at the end of the linux line, add "rw init=/bin/bash", at the end of the line then press F10 to boot:
       linux   /boot/vmlinuz-4.19.283-3.ph3-esx root=UUID=####-####-####-####-############ rw init=/bin/bash
   
   
9. Check the status of the vmware-system-user's expiration:
   chage -l vmware-system-user
   
   
10. Set the vmware-system-user to never expire:
    chage -m 0 -M -1 vmware-system-user
    
    
11. Confirm that the vmware-system-user is no longer expired:
    chage -l vmware-system-user
    
    
12. Reboot the VM by running:
    reboot -f

 

 

 

Additional Information

This needs to be done on all the control plane nodes in the Guest Cluster, one at a time. After performing the steps on the first control plane node, please wait for it to restart and obtain an IP address. Post that, proceed with the remaining control plane nodes one at a time.

Bypass Containerized Permissions by accessing the vm directly through the ESXi interface. In some cases, editing the Boot Delay of a VM is necessary, typically when adding resources to a powered-off VM.