Aria Automation Appliance root login is inaccessible (CLI and console)

Products

• VMware Aria Suite
• VMware vRealize Automation 8.x

Issue/Introduction

Unable to log into Aria Automation appliance with root account using Command Line or vSphere Client Console.

When attempting to log in to SSH using the root account gets "Access Denied".

When attempting to log into the Console from vCenter gets "Permission Denied".

Resetting the password via the grub menu does not work after following How do I reset the root password for VMware Aria Automation

Environment

Aria Automation (vRA) 8.16 and earlier

Cause

This issue is caused by the /var/log directory running out of disk space

Resolution

To gain access to the appliance with SSH or console, access the grub menu as per How do I reset the root password for VMware Aria Automation, then, instead of updating the password, edit the /etc/pam.d/system-auth to temporarily bypass the pam settings:

1. 1. Access the grub menu per How do I reset the root password for VMware Aria Automation
      
      
   2. Instead of resetting the password, open /etc/pam.d/system-auth in a text editor.

   3. Comment out the following line by adding a # in front of it (order of parameters may vary, which is fine):

      Before: auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
      After:  #auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog

   4. Save and close the file.
      
      
   5. Type sync and press Enter to flush the data to disk.
      
      
   6. Type umount / and press Enter.

   7. Type reboot -f and press Enter.

      Note: If the reboot command fails, restart the Virtual Machine through vSphere.

   8. In the vSphere Client, reopen the console of the desired node and login using root.

   9. Now that you can log in, the /etc/pam.d/system-auth file can be reverted and the space issue can be investigated.

   10. Open /etc/pam.d/system-auth in a text editor.

   11. Uncomment the following line by adding a # in front of it (order of parameters may vary, which is fine): 

       Before: #auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
       After: auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog

   12. Save and close the file.

   13. Run df -h to verify the directory that is full, it should be /var/log/.  

   14. Clean up large files that have caused the directory to fill up.

       find /var/log/ -mount -type f -mtime +1 -exec echo {} \; -exec rm {} \; 2>&1 | tee /tmp/files_removed.txt

Additional Information

When changing the password after accessing the grub menu, passwd and pam tally commands may not work correctly until space is cleared. 

Running "pam_tally2 --user root --reset" or "faillock --user root --reset"  after editing /etc/pam.d/system-auth and gaining access to the appliance via SSH or console will give a "No Space Left on Device" error. 

Note : pam_tally is deprecated from PhotonOS 4. From Aria Automation 8.16.1 and later, the PhotonOS version is 4.

           PhotonOS 4 replaces pam_tally with faillock.