vCenter trustmanagement service fails to start with "Operation timed out" or "no healthy upstream"
search cancel

vCenter trustmanagement service fails to start with "Operation timed out" or "no healthy upstream"

book

Article ID: 378362

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Accessing the vSphere Client UI results in a "no healthy upstream" error, and the VAMI (port 5480) may return a "503 - OK" or "Service Unavailable" message.
  • Critical services such as vmware-stsd, vmware-trustmanagement, and vsphere-ui are in a stopped state.
  • Manual attempts to start the services via service-control --start --all fail with an "Operation timed out" error.
    • <DATE_TIME> ERROR service-control Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start trustmanagement, sps, vapi-endpoint, sts services. Error: Operation timed out

  • From /var/log/vmware/trustmanagement-svcs.log, we found this error "Native platform error [code: 40712]" message and backtrace.
    • Caused by: com.vmware.identity.interop.NativeCallException: Native platform error [code: 40712][null][null]
              at com.vmware.identity.interop.registry.RegistryAdapter.checkNativeErrorCode(RegistryAdapter.java:1334) ~[vmware-identity-platform.jar:?]
              at com.vmware.identity.interop.registry.RegistryAdapter.openKey(RegistryAdapter.java:427) ~[vmware-identity-platform.jar:?]
              at com.vmware.identity.interop.registry.LinuxRegistryAdapter.openKey(LinuxRegistryAdapter.java:77) ~[vmware-identity-platform.jar:?]
              at com.vmware.identity.interop.registry.LinuxRegistryAdapter.openRootKey(LinuxRegistryAdapter.java:88) ~[vmware-identity-platform.jar:?]
              at com.vmware.identity.idm.server.config.IdmServerConfig.<init>(IdmServerConfig.java:372) ~[vmware-identity-idm-server.jar:?]
              at com.vmware.identity.idm.server.config.IdmServerConfig.<clinit>(IdmServerConfig.java:145) ~[vmware-identity-idm-server.jar:?]
              at com.vmware.identity.idm.server.config.ConfigStoreFactory.<init>(ConfigStoreFactory.java:34) ~[vmware-identity-idm-server.jar:?]
              at com.vmware.identity.idm.server.IdentityManager.<init>(IdentityManager.java:401) ~[vmware-identity-idm-server.jar:?]
              at com.vmware.identity.idm.server.IdentityManager.getIdmInstance(IdentityManager.java:12878) ~[vmware-identity-idm-server.jar:?]
              at com.vmware.identity.idm.client.CasIdmClient.<init>(CasIdmClient.java:154) ~[vmware-identity-idm-client.jar:?]
              at com.vmware.identity.idm.client.CasIdmClient.<init>(CasIdmClient.java:141) ~[vmware-identity-idm-client.jar:?]
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_345]

  • From /var/log/vmware/sso/sts-health-status.log
    • <DATE_TIME> - __main__ - INFO - Stderr: Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true -Dorg.apache.xml.security.ignoreLineBreaks=true
      log4j:WARN No appenders could be found for logger (com.vmware.identity.interop.NativeLibraryPreloader).
      log4j:WARN Please initialize the log4j system properly.
      Exception in thread "main" com.vmware.identity.interop.NativeCallException: Native platform error [code: 40712][null][null]
              at com.vmware.identity.interop.registry.RegistryAdapter.checkNativeErrorCode(RegistryAdapter.java:1334)
              at com.vmware.identity.interop.registry.RegistryAdapter.openKey(RegistryAdapter.java:427)
              at com.vmware.identity.interop.registry.LinuxRegistryAdapter.openKey(LinuxRegistryAdapter.java:77)
              at com.vmware.identity.interop.registry.LinuxRegistryAdapter.openRootKey(LinuxRegistryAdapter.java:88)
              at com.vmware.identity.configure.HostnameReader.read(HostnameReader.java:40)
              at com.vmware.identity.configure.HostnameReader.readPortNumber(HostnameReader.java:66)
              at com.vmware.identity.health.StsHealthStatus.main(StsHealthStatus.java:22)

  • When you check registry key using the following command with trustmanagement user like below, there is error with 40712 code.
    • sudo -u trustmanagement /opt/likewise/bin/lwregshell list_keys [HKEY_THIS_MACHINE\\Services\\lsass\\Parameters\\Providers\\ActiveDirectory]
    • Output:
      • lwregshell (error = 40712 - LWREG_ERROR_INVALID_ACCESS_TOKEN)

Environment

  • VMware vCenter Server Appliance 8.x

Cause

  • The root cause of the service startup failure is an operational dependency within the Likewise identity architecture (lwsmd / lwregd) that forces external domain validation for local non-root service users during local registry queries.
  • When a non-root internal service account such as trustmanagement attempts to query the local registry (HKEY_THIS_MACHINE), the database provider tries to generate a security access token.
  • This specific code path requests Active Directory object identification via lsassd.
  • If the configured Active Directory domain controller is offline or completely unreachable during vCenter Server startup, lsassd fails the RPC transaction and returns an offline domain error.
  • This aborts token generation, resulting in native platform registry error code 40712, which starves dependent services of configuration data and prevents them from starting.

Resolution

To resolve this issue, unjoin the vCenter Server from the Active Directory domain to remove the external dependency during service startup.

  • Log in to the vCenter Server Appliance as root via SSH or the VM console.
  • Verify the current domain status: 
    • /opt/likewise/bin/domainjoin-cli query
  • Unjoin the vCenter Server from the domain:
    • /opt/likewise/bin/domainjoin-cli leave
  • Reboot the vCenter Server appliance: 
    • reboot
  • Verify that all services start successfully and the UI is accessible.
  • Rejoin the domain if required, ensuring the Domain Controller is fully reachable.

 

Powered by Wolken Software