Changing the default password hashing algorithm from md5 to sha512 in ESXi / ESX 4.x

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on changing the default password hashing algorithm from md5 to sha512 on ESX 4.x hosts.

Symptoms:

Environment

VMware ESXi 4.1.x Installable
VMware ESXi 4.0.x Installable
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x

Resolution

Setting the ESXi host to use SHA512 encryption algorithm

1. Log in to the ESXi host using the Local Tech Support Mode or SSH.
   
   Note: Ensure that the appropriate support mode is enabled in Direct Console > Customize System (F2) > Troubleshooting Options.
   
   
2. Back up the /etc/shadow and /etc/pam.d/system-auth files.
3. Run these commands to assign edit permission to the system-auth file.
   
   #cd /etc/pam.d
   #chmod 644 system-auth
   
   
4. Open the system-auth file using a text editor.
5. Search for the password sufficient entry in the file, similar to:
   
   password sufficient /lib/security/$ISA/pam_unix.so use_authtok nullok shadow
   
   
6. Replace the existing hash key ( md5, des, or sha256) with sha512 or append sha512, if there is no existing key. For example:
   
   password sufficient /lib/security/$ISA/pam_unix.so use_authtok nullok shadow sha512
   
   
7. Run this command to restore the system-auth file permissions:
   
   #chmod 444 pam.d/system-auth
   
   
8. If you want all passwords to be in the new hash format, you must renew the passwords for all users.
   
   For example, to change the hash format for the root password and to verify if the encryption algorithm is updated:
   
   
   1. Run this command to change the root password:
      
      #passwd root
      
      
   2. Open the /etc/shadow file and verify the if the encryption algorithm is updated.
      
      To verify the encryption algorithm, run this command:
      
      #cat /etc/shadow
      
      You see an output similar to:
      
      root:$6$BpPe7UrJ$oLexOiRNefHHDsfVETKTL0:13358:0:99999:7:::
      
      The tag $6$ in the output indicates that a sha512 encryption algorithm is used.

Note: In ESXi, changes made to the /etc/pam.d/system-auth file to edit password settings do not persist across system reboots.

This issue is fixed in ESXi 5.0.

Setting the ESX host to use SHA512 encryption algorithm

1. Log in to the ESX host using the console or SSH.
   
2. Back up these files:
   
   
   • /etc/pam.d/system-auth-*
   • /etc/libuser.conf
   • /etc/login.defs
   • /etc/shadow
     
     
3. Run these commands to assign edit permissions to the system-auth , libuser.conf , and login.defs files:
   
   #cd /etc
   #chmod 644 pam.d/system-auth-*
   #chmod 644 libuser.conf
   #chmod 644 login.defs
   
   
4. Open the /etc/pam.d/system-auth-generic file using a text editor.
   
5. Search for the password sufficient entry in the file, similar to:
   
   password sufficient /lib/security/$ISA/pam_unix.so use_authtok nullok shadow
   
   

6. Replace the existing hash key ( md5, des, or sha256) with sha512 or append sha512, if there is no existing key. For example:
   
   password sufficient /lib/security/$ISA/pam_unix.so use_authtok nullok shadow sha512
   

7. Open the /etc/pam.d/system-auth-local file using a text editor.
   
8. Repeat Steps 5 and 6 for the /etc/pam.d/system-auth-local file.
   
9. Open the /etc/libuser.conf file using a text editor.
   
10. Change this entry:
    
    crypt_style = md5
    
    to
    
    crypt_style = sha512
    
    
11. Open the /etc/login.defs file using a text editor.
    
12. Change this entry:
    
    MD5_CRYPT_ENAB yes
    
    to
    
    MD5_CRYPT_ENAB no
    
    
13. Change this entry:
    
    ENCRYPT_METHOD DES
    
    to
    
    ENCRYPT_METHOD SHA512
    
    Note: If the MD5_CRYPT_ENAB and ENCRYPT_METHOD entries are not available in the /etc/login.defs file, add them at the end of the file:
    
    MD5_CRYPT_ENAB no
    ENCRYPT_METHOD SHA512
    
    
14. Run these commands to restore the permissions on the system-auth , libuser.conf , and login.defs files:
    
    #chmod 444 pam.d/system-auth-*
    #chmod 444 libuser.conf
    #chmod 444 login.defs
    
    
15. If you want all passwords to be in the new hash format, you must renew the passwords for all users.
    
    For example, to change the hash format for the root password and to verify if the encryption algorithm is updated:
    
    
    1. Run this command to change the root password:
       
       #passwd root
       
       
    2. Open the /etc/shadow file and verify the if the encryption algorithm is updated.
       
       To verify the encryption algorithm, run this command:
       
       #cat /etc/shadow
       
       You see an output similar to:
       
       root:$6$zR1rwYqO$aP/hDfnYQQ1oJVigYbG3O0:14833:0::7:::
       
       The tag $6$ in the output indicates that a sha512 encryption algorithm is used.

Note: Changing the hash method does not rehash existing passwords. It only impacts the format in which future passwords are stored when new accounts are created and passwords of existing accounts are modified.

Additional Information

There are two other weaker password hash methods: DES and SHA256. To use these hash methods, you must update the hash methods to des or sha256 in the appropriate files. Ensure to use DES and SHA256 (upper case) in the login.defs file.

Note: VMware does not recommend using these weaker password has methods.

The following table lists the various encryption methods and their tags as seen in the /etc/shadow file:

Encryption Method	Tag as seen in the /etc/shadow file
DES	13-char
MD5	$1$
SHA256	$5$
SHA512	$6$