Unable to launch SDDC Manager SSH Session due to Disk Space or Password Expiration

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Users are unable to establish a secure shell (SSH) connection to the SDDC Manager appliance. This issue is typically characterized by connection rejection or "Authentication token manipulation" errors when attempting to modify credentials. Symptoms include:

 

 

• The /dev/mapper/vg_system-lv_root or /var/log partitions reaching 100% utilization. 
• SSH daemon failing to spawn new sessions.
• Expired vcf or root account passwords.

 

Environment

VMware Cloud Foundation  5.x

 

Cause

There are two primary root causes for this behavior:

1. Credential Expiration: The vcf or root account passwords have reached their expiration limit, preventing authentication.

2. Disk Space Exhaustion: The root partition (/) or log partition is full. This prevents the Pluggable Authentication Modules (PAM) from writing necessary temporary files or updating the shadow file, often resulting in pam_tally2 errors or general authentication failures.

Resolution

If you cannot SSH into the SDDC Manager, you must access the appliance via Virtual Machine Console.

Check Partition Usage:
Execute the following command to verify disk space:

df -h

Review the output for any partition at 100% capacity, specifically /dev/mapper/vg_system-lv_root.

Clear System Logs (If Disk is Full):
If the root partition is full, reclaim space by vacuuming the journal logs:

journalctl --vacuum-size=500M

Reset Account Passwords:
Once disk space is available, proceed with resetting the expired passwords. Refer to the following procedure for password recovery:

KB 323984: Resetting the root password for the SDDC Manager appliance.

Verify SSH Service:
After clearing space and updating passwords, ensure the SSH service is running:

systemctl status sshd

If necessary, restart the service:

systemctl restart sshd