Unable to add Active Directory permissions after rebooting an ESXi 6.0 host

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
After rebooting an ESXi 6.0 GA (2494585), Patch Release ESXi600-201504001 (2615704), or Patch Release ESXi600-201505001 (2715440) which has been added to your Active Directory domain, you experience the following symptoms:

When attempting to add users or groups to the ESXi Permissions tab from the vSphere Client, you observe:
- This error is displayed:
  
  Error:A general system error occurred: Error accessing directory: Can't bind to LDAP server for domain: <Domain Name>
  
  Error Stack
  
  Call "UserDirectory.RetrieveUserGroups" for object "ha-user-directory" on ESXi "<ESXi_Host_FQDN>" failed.
- In the /var/log/vmware/hostd.log you observe the following after attempting to add an Active Directory user or group:
  <YYYY-DD-MMT>T<time> info hostd[291C1B70] [Originator@6876
  </time>sub=SysCommandPosix opID=32B1FD0C-00000053-9bb8 user=root] ForkExec(/bin/kinit)
  36333
  <YYYY-DD-MMT>T<time></time> error hostd[291C1B70] [Originator@6876
  sub=UserDirectory opID=32B1FD0C-00000053-9bb8 user=root] LDAP error code: 48
  (Inappropriate authentication)
  <YYYY-DD-MMT>T<time></time> error hostd[291C1B70] [Originator@6876 sub=Default
  opID=32B1FD0C-00000053-9bb8 user=root] Error accessing directory: Can't bind to
  LDAP server for domain: <Domain Name>
  <YYYY-DD-MMT>T<time></time> info hostd[291C1B70] [Originator@6876 sub=Default
  opID=32B1FD0C-00000053-9bb8 user=root] AdapterServer caught exception:
  vmodl.fault.SystemError
  <YYYY-DD-MMT>T<time></time> info hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi
  opID=32B1FD0C-00000053-9bb8 user=root] Activation
  [N5Vmomi10ActivationE:0x29a053c8] : Invoke done [retrieveUserGroups] on
  [vim.UserDirectory:ha-user-directory]
  <YYYY-DD-MMT>T<time>
  </time> verbose hostd[291C1B70] [Originator@6876
  sub=Solo.Vmomi opID=32B1FD0C-00000053-9bb8 user=root] Arg domain:
  --> "<Domain Name>"
  
  ...
  
  <YYYY-DD-MMT>T<time> info hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi
  </time>opID=32B1FD0C-00000053-9bb8 user=root] Throw vmodl.fault.SystemError
  <YYYY-DD-MMT>T<time></time> info hostd[291C1B70] [Originator@6876 sub=Solo.Vmomi
  opID=32B1FD0C-00000053-9bb8 user=root] Result:
  --> (vmodl.fault.SystemError) {
  --> faultCause = (vmodl.MethodFault) null,
  --> reason = "Error accessing directory: Can't bind to LDAP server for
  domain: <Domain Name>",
  --> msg = ""
  --> }
  <YYYY-DD-MMT>T<time></time> info hostd[28640B70] [Originator@6876 sub=Solo.Vmomi
  opID=8ca29ba3 user=root] Activation [N5Vmomi10ActivationE:0x27c1cbf0] : Invoke
  done [waitForUpdatesEx] on
  [vmodl.query.PropertyCollector:ha-property-collector]
  YYYY-DD-MMT<time>Z verbose hostd[28640B70] [Originator@6876
  </time>sub=Solo.Vmomi opID=8ca29ba3 user=root] Arg version:
  --> "2"
  <YYYY-DD-MMT>T<time></time> verbose hostd[28640B70] [Originator@6876
  sub=Solo.Vmomi opID=8ca29ba3 user=root] Arg options:
  --> (vmodl.query.PropertyCollector.WaitOptions) {
  --> maxWaitSeconds = 600,
  --> maxObjectUpdates = 100
  --> }
  <YYYY-DD-MMT>T<time></time> info hostd[28640B70] [Originator@6876 sub=Solo.Vmomi
  opID=8ca29ba3 user=root] Throw vmodl.fault.RequestCanceled
  YYYY-DD-MMT<time>Z info hostd[28640B70] [Originator@6876 sub=Solo.Vmomi
  </time>opID=8ca29ba3 user=root] Result:
  --> (vmodl.fault.RequestCanceled) {
  --> faultCause = (vmodl.MethodFault) null,
  --> msg = ""
  --> }
When attempting to log into the ESXi host using Use Windows sessions credentials from the vSphere Client, you observe:

A general system error occurred: gss_acquire_cred failed
I

Environment

VMware vSphere ESXi 6.0

Resolution

This is a known issue affecting ESXi 6.0.

This issue is resolved in VMware ESXi 6.0, Patch Release ESXi600-201507001. For more information, see VMware ESXi 6.0, Patch Release ESXi600-201507001 (2111982). Download the latest version at VMware downloads.

To work around this issue, perform the following depending on the scenario you are experiencing.

Before applying ESXi 6.0 patches to a host:

Connect to the ESXi host with an SSH session using root credentials. For more information, see Using ESXi Shell in ESXi 5.x and 6.0 (2004746).
Backup the existing krb5.keytab file:

cp /etc/krb5.keytab /vmfs/volumes/<persistent_datastore>/
Verify the MD5 Checksum of the krb5.keytab file:

md5sum /etc/krb5.keytab

You see a checksum output similar to:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
Upgrade the ESXi 5.x host to ESXi 6.0
After the upgrade is completed, copy the backup copy of the krb5.keytab file back into the /etc/ directory:

cp /vmfs/volumes/persistent_datastore/krb5.keytab /etc/krb5.keytab
Ensure the MD5 Checksum of the krb5.keytab file is identical to the pre-upgraded ESXi host from Step 2.

md5sum /etc/krb5.keytab

You see a checksum output similar to this:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
After completing, ensure that you are able to log into the ESXi host with Active Directory credentials.

Before rebooting your ESXi 6.0 host:

Connect to the ESXi host with an SSH session using root credentials. For more information, see Using ESXi Shell in ESXi 5.x and 6.0 (2004746).
Backup the existing krb5.keytab file.

cp /etc/krb5.keytab /vmfs/volumes/persistent_datastore/
Verify the MD5 Checksum of the krb5.keytab file

md5sum /etc/krb5.keytab

You see a checksum output similar to:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
Reboot the ESXi host.
After the upgrade has completed, copy the backup copy of the krb5.keytab file back into the /etc/ directory:

ccp /vmfs/volumes/persistent_datastore/krb5.keytab /etc/krb5.keytab
Ensure the MD5 Checksum of the krb5.keytab file is identical to the output in Step 2.

md5sum /etc/krb5.keytab

You see a checksum output similar to:

beb11f1219126c191fcd66736bbff778 /etc/krb5.keytab
After completing, ensure that you are able to log into the ESXi host with Active Directory credentials.

In the event the the krb5.keytab file has been lost:

In the event the krb5.keytab file has been lost, rejoin the ESXi 6.0 host to the Active Directory domain to recreate the file. For more information, see Add a Host to Directory Service Domain in the vSphere Security Guide. If this option is used, any krb5.keytab file that has been backed up from the methods listed above should not be reused.

Additional Information

Using ESXi Shell in ESXi 5.x and 6.x
VMware ESXi 6.0, Patch Release ESXi600-201507001
ESXi 6.0 をインストールまたはこれにアップグレードすると Active Directory 権限を追加できない
安装或升级到 ESXi 6.0 之后，无法添加 Active Directory 权限