An error occurred while starting service vsan-health
search cancel

An error occurred while starting service vsan-health

book

Article ID: 392832

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

  • In the vmware-vsan-health-service.log there will be lots of "unauthenticated" messages for localhost:8085/sdk.

    vmware-vsan-health-service.log:
    YYYY-MM-DDTHH:MM:SS ERROR vsan-mgmt[4057640] [VsanVcCapabilitySystemImpl::RetrieveHostCapabilities opID=DgqPqx-T1] Fail to get host capability for 'vim.HostSystem:host-23'
    Traceback (most recent call last):
      File "bora/vsan/capability/vpxd/pyMoVsan/VsanVcCapabilitySystemImpl.py", line 311, in RetrieveHostCapabilities
      File "bora/vsan/perfsvc/VsanHostHelper.py", line 135, in GetHostHealthSystem
      File "bora/vsan/perfsvc/VsanHostHelper.py", line 107, in _GetHostEntry
      File "bora/vsan/perfsvc/VsanHostHelper.py", line 172, in _ConnectToHostSystem
      File "bora/vsan/perfsvc/VsanHostHelper.py", line 359, in _GetStubAdapter
    PyCppVmomi.vim.fault.NotAuthenticated: (vim.fault.NotAuthenticated) {
      msg = 'Received SOAP response fault from [<<cs p:00007feb3c2fe210, TCP:localhost:8085>, /sdk>]: GetSummary\nThe session is not authenticated.',
      object = 'vim.HostSystem:host-xx',
      privilegeId = ''
    }


  • TCP 8085 is vpxd, so the service is observed to get this error back when trying to initiate a call about the host in vpxd.


  • At the same time a flood of auth errors for SSO observed in the vpxd logs:

    vpxd*
    YYYY-MM-DDTHH:MM:SS error vpxd[06088] [Originator@6876 sub=SsoWrapper opID=wcp-licenseExpirationRefreshMonitor-d9] [AcquireToken] AcquireToken exception:
     N9SsoClient22CommunicationExceptionE(Server returned error: 503 Service Unavailable)
    --> [context]zKq7AVECAQAAAA8jcwEgdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwAYq0oBTT4CbGlic3NvY2xpZW50LnNvAAGMSwIBbWsDAsVuAGxpYnNzb3V0aWwuc28AAghwAIOx5tABdnB4Z
    ACDWZksAoNlpSwCg9amLAKDtqosAgTX2hpsaWJ2bW9taS5zbwCF3pspAWxpYnZpbS10eXBlcy5zbwCDq0gtAoPpWy0Cg2RvLAKFQgk1AYNtFYICBE11HIOk718Cg+yggAKDN7GAAoMrxH8CgxqbgAIABOw3ABd
    FOAC7D1EGsI4AbGlicHRocmVhZC5zby4wAAff+g9saWJjLnNvLjYA[/context]
    YYYY-MM-DDTHH:MM:SS error vpxd[06088] [Originator@6876 sub=licenseServiceProxy opID=wcp-licenseExpirationRefreshMonitor-d9] Authentication failed, stubAdapter: <<<cs p:00007fdea80021d0, TCP:localhost:1080> >, /ls/sdk>, error: N9SsoClient22CommunicationExceptionE(Server returned error: 503 Service Unavailable)
    --> [context]zKq7AVECAQAAAA8jcwEgdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwAYq0oBTT4CbGlic3NvY2xpZW50LnNvAAGMSwIBbWsDAsVuAGxpYnNzb3V0aWwuc28AAghwAIOx5tABdnB4Z
    ACDWZksAoNlpSwCg9amLAKDtqosAgTX2hpsaWJ2bW9taS5zbwCF3pspAWxpYnZpbS10eXBlcy5zbwCDq0gtAoPpWy0Cg2RvLAKFQgk1AYNtFYICBE11HIOk718Cg+yggAKDN7GAAoMrxH8CgxqbgAIABOw3ABdFOAC7D1EGsI4AbGlicHRocmVhZC5zby4wAAff+g9saWJjLnNvLjYA[/context]
    YYYY-MM-DDTHH:MM:SS info vpxd[06088] [Originator@6876 sub=vmomi.soapStub[7] opID=wcp-licenseExpirationRefreshMonitor-d9] SOAP request returned HTTP failure; <<cs p:00007fdea80021d0, TCP:localhost:1080>, /ls/sdk>, method: isFeatureAvailable; code: 500(Internal Server Error); fault: (vmodl.fault.SecurityError) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = <unset>
    -->    msg = "Received SOAP response fault from [<<cs p:00007fdea80021d0, TCP:localhost:1080>, /ls/sdk>]: isFeatureAvailable
    --> Authentication result: Missing session auth data"
    --> }


  • vmware-stsd will occasionally stops:

    service-control.log:
    YYYY-MM-DDTHH:MM:SS INFO service-control Running:
     applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsm vsphere-ui vstats vtsdb wcp
    YYYY-MM-DDTHH:MM:SS INFO service-control Stopped:
     vmcam vmonapi vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-stsd vmware-vcha vmware-vsan-health


  • In the log file hs_err_sts_pid3798107.log, the below is observed for STS:

    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (malloc) failed to allocate 219536 bytes for Chunk::new

Environment

VMware vCenter Server 8.x

Cause

Issue caused by insufficient runtime memory allocated to STS service

Resolution

Take snapshot of the vCenter before you proceed with the below:

  1. Verify the amount memory assigned to vmware-stsd: cloudvm-ram-size -l | grep -i vmware-stsd

  2. Increase the amount of memory assigned to the vmware-stsd services by running the following command: cloudvm-ram-size -C xxx vmware-stsd
    Where xxxx is in MB and remember 1G= 1024 MB
    So if xxxx is less than 1024 MB assign

    Example: cloudvm-ram-size -C 1024 vmware-stsd

    or if xxx > 1024, you can assign 2048

  3. Restart services:
    service-control --restart vmware-stsd
    service-control --restart vsan-health
Powered by Wolken Software