Root login attempt fails with "Permission denied, please try again"

Products

• VCF Operations

Issue/Introduction

You experience the following symptom when attempting to access the appliance via SSH or the console:

• Root login attempt fails with the following error:

Permission denied, please try again.

Environment

Lifecycle manager  8.18.0

Cause

Drive space is full. Storage exhaustion (ENOSPC) on the root (/) or /var/log partitions directly disrupts Pluggable Authentication Module (PAM) routines and the SSH daemon (sshd).

If the system cannot write session tokens, audit records, or authentication logs, it forcefully rejects SSH connections with the "Permission denied" error. Furthermore, if the disk is at 100% capacity during an initial password reset attempt, the write operation to /etc/shadow fails silently, meaning the new password hash is never committed to disk.

Resolution

1. Gain Emergency Console Access

• Attempt to log in via the vCenter Web Console or Remote Console (VMRC).
• If the console login succeeds, proceed to Step 2.
• If the console login accepts the password but immediately drops back to the login prompt, the disk is entirely full. You must reboot the appliance, access the GNU GRUB Edit Menu, append rw init=/bin/bash to the boot line, and press F10 to force a root shell.

2. Validate Capacity

• Identify the exhausted mount points by running the following command:

df -h

• Check for the / or /var/log partitions showing 100% usage.

3. Reclaim Space Safely

• Locate the largest files consuming space in the log directory:

find /var/log -type f -size +50M -exec ls -lh {} \;

• To reclaim space from active log files, use truncation rather than deletion. Using the rm command on an actively written log file leaves an orphaned file handle holding the disk space in memory until the service is restarted. Truncate the file using:

truncate -s 0 /var/log/<target_file>.log

• Clear outdated systemd journal logs to reclaim immediate capacity:

journalctl --vacuum-size=100M

4. Reapply the Password Hash

• Once a minimum of 5-10% capacity is reclaimed on the root partition, you must reset the password again to ensure the hash successfully commits to the shadow file:

passwd root

5. Restore Services

• If you accessed the system via the GRUB init=/bin/bash method, reboot the appliance to restore normal operations.