Unknown Services: When running "get cluster status" a number of services are reported as UNKNOWN (SEARCH, APPLIANCE proxy, and SHA).

Products

• VMware NSX

Issue/Introduction

• In the NSX_UI under System --> Appliances --> VIEW DETAILS and when reviewing the Service Status there is no status provided for the services 
  
  
• A syslog server has been configured on the NSX Manager
• Log lines similar to the below are encountered on the NSX Manager in /var/log/syslog

  <timestamp> NSX 1738 - [nsx@4413 comp="nsx-manager" subcomp="sha" username="nsx-sha" level="WARNING" s2comp="rpc-client" invalid="true"] Event RpcStubCreateEvent(stub=ShaMetricService_Stub, retry=False): Exception in initializing RPC client for ShaMetricService_Stub: LmMetricRpcStub Onboard fails for APH e9bf98ef-####-####-####-################, retry after 59.69005503965839 seconds

• When logged into the NSX Manager as admin and running the command  "get cluster status"


  Service Name: SEARCH

  Nodes:
    UUID                                       IPV4             IPV6                                       STATUS               FQDN
    e2ec3242-###-####-####-#######      ###.###.#.##     not configured                                   UNKNOWN               nsx_3.example.com

      c52f3242-###-####-####-#######      ###.###.#.##     not configured                                   UNKNOWN                nsx_1.example.com
 
    5a6e3242-###-####-####-#######      ###.###.#.##     not configured                                   UNKNOWN               nsx_2.example.com

  Service Name: APPLIANCE_PROXY

  Nodes:
    UUID                                       IPV4             IPV6                                        STATUS               FQDN
    e2ec3242-###-####-####-#######       ###.###.#.##     not configured                                   UNKNOWN               nsx_3.example.com

      c52f3242-###-####-####-#######       ###.###.#.##     not configured                                   UNKNOWN               nsx_1.example.com

      5a6e3242-###-####-####-#######       ###.###.#.##     not configured                                   UNKNOWN               nsx_2.example.com

  Service Name: SHA

  Nodes:
    UUID                                       IPV4             IPV6                                        STATUS               FQDN
    e2ec3242-###-####-####-#######       ###.###.#.##     not configured                                   UNKNOWN               nsx_3.example.com

      c52f3242-###-####-####-#######       ###.###.#.##     not configured                                   UNKNOWN               nsx_1.example.com

      5a6e3242-###-####-####-#######       ###.###.#.##     not configured                                   UNKNOWN               nsx_2.example.com

Environment

NSX 9.0

Cause

 SHA receives the onboard response and try to wake up the sending thread before it entering the wait state, and then the sending thread waits till timeout and the onboard failure occurs.

Resolution

This is a known issue impacting VMware NSX.

Workaround 

Remove the syslog server configuration for the NSX Manager nodes and then restart the SHA service

restart service sha