VMware Identity Management Service fails after accessing the Platform Services Controller Appliance 6.0

Products

• VMware Aria Suite
• VMware vCenter Server

Issue/Introduction

Symptoms:
When a Platform Services Controller Appliance 6.0 is placed under heavy log-in load, you may experience these symptoms:

• VMware Identity Management Service (vmware-sts-idmd) will crash or become unresponsive
• Logins to vCenter Server or second party components fail.
• In the /var/log/vmware/sso/hs_err_idm_pidXXXX.log file, you see a JVM crash report. Towards the top of the stack trace (Java frames) section of the report, you see entries similar to:
  
  Note: In the hs_err_idm_pidXXXX.log file, pidXXXX is the PID of the vmware-sts-idmd service.
  
  Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
  J 506 com.sun.jna.Native.invokeInt(JI[Ljava/lang/Object;)I (0 bytes) @ 0x00007f6af7e48a38 [0x00007f6af7e489e0+0x58]
  J 292 C2 com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;Z)Ljava/lang/Object; (1154 bytes) @ 0x00007f6af7de6d80 [0x00007f6af7de6aa0+0x2e0]
  J 162 C2 com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object; (526 bytes) @ 0x00007f6af7db98e8 [0x00007f6af7db8d20+0xbc8]
  J 291 C2 com.sun.jna.Library$Handler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (345 bytes) @ 0x00007f6af7df79dc [0x00007f6af7df7580+0x45c]
  j com.sun.proxy.$Proxy13.IDMAuthenticateUser(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Lcom/sun/jna/ptr/PointerByReference;)I+29
  j com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.AuthenticateByPassword(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Lcom/vmware/identity/interop/idm/UserInfo;+35
   
• In the /var/log/vmware/sso/vmware-sts-idmd.err log file, you see entries similar to:
  
  ======= Backtrace: =========
  /lib64/libc.so.6(+0x79088)[0x7fc367a7b088]
  /lib64/libc.so.6(cfree+0x6c)[0x7fc367a800cc]
  /opt/vmware/lib64/libcrypto.so.1.0.1(CRYPTO_free+0x1d)[0x7fc35c3fa49d]
  /opt/vmware/lib64/libcrypto.so.1.0.1(OBJ_add_object+0x140)[0x7fc35c3fdb20]
  /opt/vmware/lib64/libcrypto.so.1.0.1(OBJ_create+0xed)[0x7fc35c3fe88d]
  /opt/likewise/lib64/krb5/plugins/preauth/pkinit.so(+0x11513)[0x7fc34f7dc513]
  /opt/likewise/lib64/krb5/plugins/preauth/pkinit.so(+0x111d5)[0x7fc34f7dc1d5]
  /opt/likewise/lib64/krb5/plugins/preauth/pkinit.so(+0xc7f1)[0x7fc34f7d77f1]
  /opt/likewise/lib64/libkrb5.so.3(+0x6b551)[0x7fc357b88551]

Cause

VMware uses the MIT Kerberos library as part of delivering SSO functionality. The MIT Kerberos library includes a plugin called pkinit which is meant to be used for pre-authentication purposes. Although this plugin is not directly leveraged by VMware, it is still loaded by the MIT Kerberos library. This plugin is unstable, and under certain conditions, can cause VMware Identity Management Service to crash.

Resolution

This is a known issue affecting the Platform Services Controller Appliance 6.0.This issue is resolved in vCenter Appliance 6.0 Update 3 available at VMware Downloads.
 

Workaround:
To work around this issue, manually move the pkinit.so files from the likewise directory:

1. Connect to the Platform Services Controller Appliance or vCenter Server with Embedded Platform Services Controller Appliance console and log in using root credentials.
2. Run this command to enable access the Bash shell:
   
   shell.set --enabled true
    
3. Type shell and press Enter.
4. Navigate to/opt/likewise/lib64/krb5/plugins/preauth/ by running this command:
   
   cd /opt/likewise/lib64/krb5/plugins/preauth/
    
5. Move the pkinit.so file to /root by running this command:
   
   mv pkinit.so /root
    
6. Restart the Platform Services Controller.

Note: Repeat the preceding steps on other Platform Services Controllers in the environment.