This issue occurs as the ESXi trust store contains a list of Certificate Authority (CA) certificates that are used to build the chain of trust when an ESXi host is a client in a TLS channel communication.
The certificates in the trust store must be with a CA bit set: X509v3 Basic Constraints: CA: TRUE. If a certificate without this bit set is passed to the trust store, for example, a self-signed certificate, the certificate is rejected. As a result, adding an ESXi host to the vCenter Server may fail.