Reset root password on all Aria Automation / Orchestrator nodes of a cluster, where there is access on a node but locked out of other nodes

Products

• VMware Aria Suite

Issue/Introduction

It is preferable to avoid a reboot, as starting up to k8s pods being "Ready" takes many minutes.

Symptoms:

When logging in to Aria Automation or Orchestrator nodes as root over SSH or on VM Console, we are unable to log in for any reason.

For example, we may be shown:

Access Denied.

Often, we have access on 1 or 2 nodes, but need to restore root access to the other(s).

Environment

Aria Automation 8.x
Aria Automation Orchestrator 8.x

Cause

There are many potential causes for root access denied on Linux:

1. Account is locked due to unsuccessful attempts or other PAM failure thresholds - (pam_tally2)
2. Password expired due to maxdays from change exceeded. Often a successful login allows (and demands) password change. 
3. The account password is locked: passwd -S shows L as 2nd value
4. The PAM settings have been changed to non-standard values for Aria Automation / Orchestrator
5. We just don't know the password

These reasons are per-node, and so in many cases we have access to 1 or 2 nodes

Resolution

Please see the numbered list of potential causes above.

1. Unlock account (PAM) on all nodes:

• You can check if the failures exceeds the maximum (default: 3 attempts needs 5 mins' wait)
  • vracli cluster exec -- bash -c 'hostname; pam_tally2 -u root'

• Unlock root on all nodes:
  • vracli cluster exec -- bash -c 'hostname; pam_tally2 -u root --reset'

2 & 3. Review these settings on all nodes:

• vracli cluster exec -- bash -c 'hostname; passwd -S root'

2. If the date in the 3rd value + the days in the 5th value has passed, you can disable this expiry:

• vracli cluster exec -- bash -c 'hostname; passwd -x -1 root'
• Once access is restored, reset maxdays expiry to 1 year:
  • vracli cluster exec -- bash -c 'hostname; passwd -x 365 root'

3. Unlock password for account if there is an L as the second value to passwd -S  :

• vracli cluster exec -- bash -c 'hostname; passwd -u root'

4. Please review all files in /etc/pam.d on customer system with reference to a fresh lab Aria Automation/Orchestrator system of same version.

• If any are different to the standard, please revert to standard values and/or review the manpages for PAM
• Get it working with standard values and customer can attempt to make unsupported modifications in a change window, at their own peril

5. Change password on all nodes to NewTemporaryPassword:

• vracli cluster exec -- bash -c 'hostname; echo -e "NewTemporaryPassword\nNewTemporaryPassword" | passwd'
• Once access is restored using NewTemporaryPassword, you can change to a desired password. Run on each node:
  • passwd

Additional Information

Impact/Risks:

This article is to avoid the impact of a reboot.

The reboot option is available here (impacting):Resetting the root password on a Photon appliance in VMware Aria Automation