Unable to reconnect the host to vCenter Server with error "A general system error occured: Failed to login with vim administrator password"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• While connecting the host to vCenter Server, it fails with the error after remaining stuck at 0%:
  
  A general system error occured: Failed to login with vim administrator password
   
• In the hostd.log file, you see entries similar to:
   

  pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_tally2(vmware-authd:auth): update (fsync) failed for /var/log/tallylog: No space left on device
pam_exec(vmware-authd:auth): /usr/lib/vmware/misc/bin/tally_vob.sh failed: exit code 1
pam_exec(vmware-authd:auth): conversation failed
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_tally2(vmware-authd:auth): update (fsync) failed for /var/log/tallylog: No space left on device
pam_exec(vmware-authd:auth): /usr/lib/vmware/misc/bin/tally_vob.sh failed: exit code 1
pam_exec(vmware-authd:auth): conversation failed
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser

  2017-05-25T02:53:41.004Z verbose hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Arg userName:
--> "vpxuser"
2017-05-25T02:53:41.004Z verbose hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Arg password:
--> (not shown)
-->
2017-05-25T02:53:41.004Z verbose hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Arg locale:
--> ""
2017-05-25T02:53:41.004Z info hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.InvalidLogin
2017-05-25T02:53:41.004Z info hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Result:
--> (vim.fault.InvalidLogin) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = ""
--> }
pam_tally2(vmware-authd:auth): user vpxuser (500) tally 6, deny 5

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.0, 7.0
VMware vCenter Server 6.0.x, 7.0.x
VMware vCenter Server Appliance 6.0.x, 7.0.X

Resolution

Based on hostd.log there are two issues here.

Section A:

• Space on device where tallylog is logged is full. In the hostd log snippet, you see this issue as:
  
  pam_tally2(vmware-authd:auth): update (fsync) failed for /var/log/tallylog: No space left on device
pam_exec(vmware-authd:auth): /usr/lib/vmware/misc/bin/tally_vob.sh failed: exit code 1
pam_exec(vmware-authd:auth): conversation failed

To resolve the issue:

1. Execute following command to check which device is full.
   
   vdf -h
    

   Output should be something like:

   Ramdisk Size Used Available Use% Mounted on
root 32M 4M 27M 12% --
etc 28M 440K 27M 1% --
opt 32M 0B 32M 0% --
var 48M 47M 4K 99% --
tmp 256M 80K 255M 0% --
iofilters 32M 0B 32M 0% --
hostdstats 2053M 106M 1946M 5% --
    

2. In above case since the var is showing full, navigate to /var/log directory.
3. Execute du -ha to see which file is occupying most space.
4. If there is any log file which is taking maximum space, move it to one of your datastores (For example, suppose if we enabled likewise logging and it took all the space).
   
   mv /var/log/likewise.log /vmfs/volumes/<Datastore name>/likewise.log
    
5. Try adding the host.

 

Section B:
 

• Second issue is password for vpxadmin user is not correctly updated with ESXi. In the hostd.log snippet, you see this issue as:
   

  pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser
pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=vpxuser

  2017-05-25T02:53:41.004Z verbose hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Arg userName:
--> "vpxuser"
2017-05-25T02:53:41.004Z verbose hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Arg password:
--> (not shown)
-->
2017-05-25T02:53:41.004Z verbose hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Arg locale:
--> ""
2017-05-25T02:53:41.004Z info hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.InvalidLogin
2017-05-25T02:53:41.004Z info hostd[B140B70] [Originator@6876 sub=Solo.Vmomi] Result:
--> (vim.fault.InvalidLogin) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = ""
--> }
pam_tally2(vmware-authd:auth): user vpxuser (500) tally 6, deny 5

To resolve the issue:

1. Stop the vpxa service running the command:
   
   /etc/init.d/vpxa stop
    
2. Delete the user vpxuser from ESXi running the command:
   
   /usr/lib/vmware/auth/bin/deluser vpxuser
    
3. Make sure that vpxadmin is deleted running the command:
   
   cat /etc/passwd
   
   Above command should not show an entry for vpxuser.
    
4. Try adding the host to vCenter Server again.
    

This will create new user vpxuser on ESXi and update its password.