VixVM_LoginInGuest failure: Authentication failure or insufficient permissions in guest operating system

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• On ESXi 4.0.x, connecting Virtual Machine's guest OS with VIX API fails with this error:
  
  Authentication failure or insufficient permissions in guest operating system.
  
  
• Virtual Machine Guest OS is Linux and has VMware tools 4.0.0, build-208167 installed on it

Environment

VMware ESXi 4.0.x Embedded
VMware ESXi 4.0.x Installable

Resolution

This is a known issue for ESXi 4.0.x caused due to VIX configuration to use the PAM libraries in the guest to perform authentication.. To resolve this issue, upgrade to ESXi to 4.1. You can download the update from the ESXi Download Center. For more information, see the ESXi 4.1 Release Notes.

Alternatively, if you do not wish to upgrade, to workaround this issue:

1. SSH to the Virtual Machine Guest OS and locate the file, /etc/pam.d/vmware-guestd, and add these lines.
   Note: If these lines are already exist, comment the existing lines and add these new lines:
   
auth sufficient /lib/security/pam_unix.so shadow nullok
account sufficient /lib/security/pam_unix.so


2. Save the file and restart the vmware-guestd service using this command:
   
   /etc/init.d/vmware-tools restart