SOS health check shows "RED" for API connectivity status on ESXi hosts

Products

• VMware SDDC Manager

Issue/Introduction

1. SOS health shows the connectivity status as RED for ESXi hosts -

Connectivity : RED
+-----+---------------------------------------
| SL# | Area 　　　　　　 | Title 　　 | State |
+-----+---------------------------------------
| 1 | ESXi :esxi_fqdn | Ping status  |GREEN  |
| |  | API Connectivity status 　     | RED   |
| |  | SSH Connectivity status 　     | RED   |

2. Retrieval of password expiry also fails for these hosts with error "Failed to get details" in SOS health check

3. sos.log show below entries-

xxxx-xx-xxTxx:xx:xx.xxx+0000 ERROR [vcf_sos] [vc.py::get_si::71::get_esxi_ssh_statusThread7] Unable to connect to host esxi_fqdn

xxxx-xx-xxTxx:xx:xx.xxx+0000 ERROR [vcf_sos] [vc.py::connect::108::get_esxi_ssh_statusThread7] Traceback (most recent call last):
  File "/opt/vmware/sddc-support/framework/../dependency/pyVpx/pyVim/connect.py", line 431, in __Login
    content = si.RetrieveContent()
  File "/opt/vmware/sddc-support/framework/../dependency/pyVpx/pyVmomi/VmomiSupport.py", line 586, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  .........
  File "/opt/vmware/sddc-support/framework/../dependency/pyVpx/pyVmomi/SoapAdapter.py", line 1132, in connect
    six.moves.http_client.HTTPSConnection.connect(self)
......
  File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

4. Issued ESXi certificates are already added to SDDC trusted roots store 

Environment

VMware Cloud Foundation 5.x

Cause

This issue happens when ESXi hosts are using VMCA signed certificates and SDDC Manager does not trust the VMCA certificate. 

Resolution

1. Retrieve the VMCA certificate from the vCenter managing the affected ESXi hosts.
   • SSH to the vCenter server
   • Copy the root.cer file from location - /var/lib/vmware/vmca/
2. Add the certificate retrieved above to SDDC Manager trust stores as per the KB article - https://knowledge.broadcom.com/external/article/316056/how-to-adddelete-custom-ca-certificates.html