Error “passwd: Authentication Token Manipulation Error” while resetting root password

Products

• VMware Aria Suite

Issue/Introduction

• Unable to reset password for root using the passwd command.
  Attempting to change the password with the command results in the following error: 
  

  "passwd: Authentication token manipulation error"

  
  
• Creating a file in the /root partition fails with Error: 
  

  No space left on device

• Same operation may work fine in another node of cluster setup.

Environment

VMware Identity Manager 3.3.x

Cause

The inode usage has reached 100%, This issue is normally caused by too many empty files being generated.

Resolution

Take SSH session to the affected node:
1) Run below script in the affected node
    for i in /*; do echo $i; find $i |wc -l; done 

2) Once identified certain directory includes large number of files, then dig down to see which base directory caused the problem:
    for i in /xxx/xxx/*; do echo $i; find $i |wc -l; done

Note: Only if you find the files are safe to be removed, run the following command to delete them.(Validate this directory from lab or other working nodes).
cd /xxx/xxx/; find . -type f | xargs rm -rf

(If the command do not work, you may have to remove the files in batches)

Validate inode usage again.
df -i