The VMware Cloud Foundation NTP health check shows a Yellow status for the PSC /vCenter, and the Compute health check shows a Red status or the PSC/vCenter

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• The /opt/vmware/sddc-support/sos --health-check reports the following:

- The NTP health check for the vCenter Server Appliance and the PSC state are Yellow.
- The Compute health check for the vCenter Server Appliance and the PSC state are RED.

• You see messages similar to the following in /var/tmp/sos.log on the SDDC Manager Controller VM 

For the NTP issue:
2018-05-24T20:35:39.739Z [DEBUG commandutils.py::run_cmds_over_ssh::377::_parallel_check_psc_ntp_configuredThread0] server: 10.23.0.122 --- stdout: [ERROR]: Failed to connect to service.
Use service-control command to manage applmgmt service

Unknown command: `com.vmware.appliance.version1.ntp.get'

For the Compute issue:-
2018-05-24T20:35:39.741Z [DEBUG commandutils.py::run_cmds_over_ssh::374::_parallel_check_psc_healthThread1] server: 10.23.0.123 --- rc: 253
2018-05-24T20:35:39.741Z [DEBUG commandutils.py::run_cmds_over_ssh::377::_parallel_check_psc_healthThread1] server: 10.23.0.123 --- stdout: [ERROR]: Failed to connect to service.
Use service-control command to manage applmgmt service
Unknown command: `system.health.get'

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 2.3.x
VMware Cloud Foundation 2.2.x

Cause

 

Resolution

This is a known issue affecting vSphere 6.5 U1. This issue is resolved in vSphere 6.5P03 and 6.7U1

Workaround:

1. SSH to the vCenter/PSC as root.
2. Run the following commands to restart the applmgmt service (VMware Appliance Management Service) 

service-control --stop applmgmt
service-control --start applmgmt

3. Repeat steps 1 and 2 on any other affected vCenter Servers or PSCs.
4.  Permanently Enable the bash shell for the root account on all PSCs and vCenters in the VMware Cloud Foundation deployment.

• ​Run the following command on the vCentetr or PSC to verify which shell is permanently enabled for the root account.

 less /etc/passwd | grep root​
 

​If the output is showing the following  root:x:0:0:root:/root:/bin/appliancesh, then the "appliancesh" shell is permanently enabled.

​If the output is showing the following root:x:0:0:root:/root:/bin/bash, then the "bash" shell is permanently enabled.
 

Note: To permanently enable the "bash" shell see Toggling the vCenter Server Appliance 6.x default shell (2100508).

Additional Information

• The SOS tool will Login to Appliance Shell  for the vCenter Server Appliance and the PSC through  SSH,  but when running the following commands as  [email protected]​ the commands will fail with error message "Unknown command" .

Commands:
com.vmware.appliance.version1.ntp.get       [To Check the NTP Status ]
system.health.get                                          [To check the Health Status]
 

• You will get an error message "Unable to authorize" when enabling shell on the vCenter Server Appliance and the PSC.

​​Example:
Command> shell
Unable to authorize
User '[email protected]' is not authorized to run this command