Cannot log in as root to Aria Automation Appliance with SSH or Console

Products

• VMware Aria Suite
• VMware vRealize Automation 8.x

Issue/Introduction

Unable to log into Aria Automation appliance with root account using SSH or vSphere Client Console

When attempting to log in to SSH the root account you get Access Denied

When attempting to log into the Console from vCenter you get Permission Denied

Resetting the password via the grub menu does not work after following https://docs.vmware.com/en/VMware-Aria-Automation/8.18/Administering-VMware-Aria-Automation/GUID-1BC4BD89-73C5-4C2A-875D-B8787418EE85.html

Environment

Aria Automation (vRA) 8.16 and earlier

Cause

This issue is caused by the /var/log directory running out of space

Resolution

To gain access to the appliance again with SSH or console again, access the grub menu as per https://docs.vmware.com/en/VMware-Aria-Automation/8.18/Administering-VMware-Aria-Automation/GUID-1BC4BD89-73C5-4C2A-875D-B8787418EE85.html and then instead of updating the password, edit the /etc/pam.d/system-auth to temporarily bypass the pam settings:

1. Access the grub menu as per https://docs.vmware.com/en/VMware-Aria-Automation/8.18/Administering-VMware-Aria-Automation/GUID-1BC4BD89-73C5-4C2A-875D-B8787418EE85.html 
   
   
2. Instead of resetting the password, open /etc/pam.d/system-auth in a text editor.
   
   
3. Comment out the following line by adding a # in front of it:
        auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
        Example: #auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
   
   
4. Save and close the file.
   
   
5. Type sync and press Enter to flush the data to disk.
   
   
6. Type umount / and press Enter.
   
   
7. Type reboot -f and press Enter.
        Note: If the reboot command fails, restart the Virtual Machine through vSphere.
   
   
8. In the vSphere Client, reopen the console of the desired node and login using root.
9. Now that you can log in, the /etc/pam.d/system-auth file can be reverted and the space issue can be investigated.
   
   
10. Open /etc/pam.d/system-auth in a text editor.
    
    
11. Uncomment the line from step 2 by removing the # in front of it.
         Example: auth    required   pam_tally2.so onerr=fail deny=3 unlock_time=900 root_unlock_time=900 file=/var/log/tallylog
    
    
12. Save and close the file.
    
    
13. run df -h to verify the directory that is full, it should be /var/log/.
    
    
14. Clean up large files that have caused the directory to fill up. 

Additional Information

When changing the password after accessing the grub menu, passwd and pam tally commands may not work correctly until space is cleared. 

Running pam_tally2 --user root --reset after editing /etc/pam.d/system-auth and gaining access to the appliance via SSH or console will give a "No Space Left on Device" error.