Unable to add Active Directory permissions after rebooting an ESXi 6.0 host

Products

VMware vSphere ESXi

Issue/Introduction

After rebooting an ESXi 6.0 GA (build: 2494585), Patch Release ESXi600-201504001 (build: 2615704), or Patch Release ESXi600-201505001 (build: 2715440) which has been added to your Active Directory domain, you may see the below issues.

When attempting to add users or groups to the ESXi 'Permissions' tab from the vSphere Client, you observe the error:

Error:A general system error occurred: Error accessing directory: Can't bind to LDAP server for domain: <Domain Name>Error Stack

Call "UserDirectory.RetrieveUserGroups" for object "ha-user-directory" on ESXi "<ESXi_Host_FQDN>" failed.

In /var/log/vmware/hostd.log, you may observe the following entries after attempting to add an Active Directory user or group:

YYYY-MM-DDTHH:MM:SS info hostd[291C1B70] [Originator@6876</time>sub=SysCommandPosix opID=32B1FD0C-00000053-9bb8 user=root] ForkExec(/bin/kinit) 36333
YYYY-MM-DDTHH:MM:SS error hostd[291C1B70] [Originator@6876sub=UserDirectory opID=32B1FD0C-00000053-9bb8 user=root] LDAP error code: 48 (Inappropriate authentication)
YYYY-MM-DDTHH:MM:SS error hostd[291C1B70] [Originator@6876 sub=Default opID=32B1FD0C-00000053-9bb8 user=root] Error accessing directory: Can't bind to LDAP server for domain: <Domain Name>
YYYY-MM-DDTHH:MM:SS info hostd[291C1B70] [Originator@6876 sub=Default opID=32B1FD0C-00000053-9bb8 user=root] AdapterServer caught exception:
vmodl.fault.SystemError
YYYY-MM-DDTHH:MM:SS info hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi opID=32B1FD0C-00000053-9bb8 user=root] Activation
[N5Vmomi10ActivationE:0x29a053c8] : Invoke done [retrieveUserGroups] on [vim.UserDirectory:ha-user-directory]
YYYY-MM-DDTHH:MM:SS verbose hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi opID=32B1FD0C-00000053-9bb8 user=root] Arg domain:
--> "<Domain Name>"
...........
YYYY-MM-DDTHH:MM:SS info hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi</time>opID=32B1FD0C-00000053-9bb8 user=root] Throw vmodl.fault.SystemError
YYYY-MM-DDTHH:MM:SS info hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi opID=32B1FD0C-00000053-9bb8 user=root] Result:
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> reason = "Error accessing directory: Can't bind to LDAP server for domain: <Domain Name>",
--> msg = ""
--> }
YYYY-MM-DDTHH:MM:SS info hostd[28640B70] [Originator@6876 sub=Solo.Vmomi opID=8ca29ba3 user=root] Activation [N5Vmomi10ActivationE:0x27c1cbf0] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]
YYYY-MM-DDTHH:MM:SS verbose hostd[28640B70] [Originator@6876</time>sub=Solo.Vmomi opID=8ca29ba3 user=root] Arg version:
--> "2"
YYYY-MM-DDTHH:MM:SS verbose hostd[28640B70] [Originator@6876 sub=Solo.Vmomi opID=8ca29ba3 user=root] Arg options:
--> (vmodl.query.PropertyCollector.WaitOptions) {
--> maxWaitSeconds = 600,
--> maxObjectUpdates = 100
--> }
YYYY-MM-DDTHH:MM:SS info hostd[28640B70] [Originator@6876 sub=Solo.Vmomi opID=8ca29ba3 user=root] Throw vmodl.fault.RequestCanceled
YYYY-MM-DDTHH:MM:SS info hostd[28640B70] [Originator@6876 sub=Solo.Vmomi</time>opID=8ca29ba3 user=root] Result:
--> (vmodl.fault.RequestCanceled) {
--> faultCause = (vmodl.MethodFault) null,
--> msg = ""
--> }

When attempting to log into the ESXi host using 'Use Windows sessions credentials' from the vSphere Client, you observe:

A general system error occurred: gss_acquire_cred failed

Environment

VMware vSphere ESXi 6.0

Resolution

This is a known issue affecting ESXi 6.0.

This issue is resolved in VMware ESXi 6.0, Patch Release ESXi600-201507001. Download the latest version at Broadcom Support portal.

To work around this issue, perform the following depending on the scenario you are experiencing.

Before applying ESXi 6.0 patches to a host:

Connect to the ESXi host with an SSH session using root credentials. For more information, see Using ESXi Shell in ESXi.
Backup the existing krb5.keytab file:

cp /etc/krb5.keytab /vmfs/volumes/<persistent_datastore>/
Verify the MD5 Checksum of the krb5.keytab file:

md5sum /etc/krb5.keytab

You see a checksum output similar to:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
Upgrade the ESXi 5.x host to ESXi 6.0
After the upgrade is completed, copy the backup copy of the krb5.keytab file back into the /etc/ directory:

cp /vmfs/volumes/persistent_datastore/krb5.keytab /etc/krb5.keytab
Ensure the MD5 Checksum of the krb5.keytab file is identical to the pre-upgraded ESXi host from Step 2.

md5sum /etc/krb5.keytab

You see a checksum output similar to this:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
After completing, ensure that you are able to log into the ESXi host with Active Directory credentials.

Before rebooting your ESXi 6.0 host:

Connect to the ESXi host with an SSH session using root credentials. For more information, see Using ESXi Shell in ESXi.
Backup the existing krb5.keytab file.

cp /etc/krb5.keytab /vmfs/volumes/persistent_datastore/
Verify the MD5 Checksum of the krb5.keytab file

md5sum /etc/krb5.keytab

You see a checksum output similar to:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
Reboot the ESXi host.
After the upgrade has completed, copy the backup copy of the krb5.keytab file back into the /etc/ directory:

ccp /vmfs/volumes/persistent_datastore/krb5.keytab /etc/krb5.keytab
Ensure the MD5 Checksum of the krb5.keytab file is identical to the output in Step 2.

md5sum /etc/krb5.keytab

You see a checksum output similar to:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
After completing, ensure that you are able to log into the ESXi host with Active Directory credentials.

In the event the the krb5.keytab file has been lost:

In the event the krb5.keytab file has been lost, rejoin the ESXi 6.0 host to the Active Directory domain to recreate the file. If this option is used, any krb5.keytab file that has been backed up from the methods listed above should not be reused.

Additional Information

Using ESXi Shell in ESXi