EAM API call fails with CertificateNotTrustedFault or EAM agent has CertificateNotTrusted issue.

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

One or more of the following symptoms can be observed:

Symptom 1: EAM API call fails with CertificateNotTrustedFault.

In the case of CertificateNotTrustedFault, a log snippet similar to this below is added to the /var/log/vmware/eam/eam_api.log file.

[YYYY-MM-DDTHH:MM:SS] | ERROR | vlsi | LocalizationFilter.java | 94 | API ERROR: EsxAgentManager.createAgency[opId=1196470828, sessionId=6823FB51]. Fault:
eam.fault.CertificateNotTrustedFault {
   url = 'https://10.185.##.###:34591/vm/vm.ovf',
}
(eam.fault.CertificateNotTrustedFault) {
   faultCause = com.vmware.vim.binding.vmodl.MethodFault: "com.vmware.eam.security.trust.NotTrusted: Suitable trust, not found!" caused by "org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)" caused by "java.security.cert.CertificateException: Unable to construct a valid chain" caused by "java.security.cert.CertPathBuilderException: Unable to find certificate chain." Please follow KB 93130,
   faultMessage = null,
   url = https://10.185.##.###:34591/vm/vm.ovf
}

Symptom 2: EAM Agent has a CertificateNotTrusted issue.

The user can detect the issue using the vSphere UI or by checking the content of the /var/log/vmware/eam/eam.log file.

The issue is presented in the ЕАМ UI (vCenter Client → Administration → vCenter Server Extensions → ESX Agent Manager → Monitor) in one of the following ways:

The issue is presented in the /var/log/vmware/eam/eam.log file with a snippet similar to one of the following:

Snippet 1:

[YYYY-MM-DDTHH:MM:SS] | DEBUG | host-82-0 | IssueHandlerBase.java | 539 | About to add unique issue: HostAgentIssueHandler:HostAgent(ID: 802a61b5-####-####-####-########f0d)
eam.issue.CertificateNotTrusted {
   time = [YYYY-MM-DDTHH:MM:SS],
   description = <unset>,
   key = 1,
   agencyName = 'test-solution-[YYYY-MM-DDTHH:MM:SS]',
   solutionId = 'eamUnitTest',
   solutionName = 'eamUnitTest',
   agency = 'Agency:e3d398f8-####-####-####-########423:null',
   agent = 'Agent:802a61b5-####-####-####-########f0d:null',
   agentName = '802a61b5-####-####-####-########f0d',
   host = 'HostSystem:host-82:3b47f7fd-####-####-####-########33a',
   hostName = '10.186.##.###',
   url = 'https://10.93.##.###:8080/vm/vm.ovf',
}

Snippet 2:

[YYYY-MM-DDTHH:MM:SS] | DEBUG | cluster-agent-1 | IssueHandlerBase.java | 539 | About to add unique issue: ClusterAgentIssueHandler:ClusterAgent(ID: 9325a7d2-####-####-####-########196)
eam.issue.cluster.agent.CertificateNotTrusted {
   description = <unset>,
   time = [YYYY-MM-DDTHH:MM:SS],
   key = 1,
   agencyName = 'test-solution-[YYYY-MM-DDTHH:MM:SS]',
   solutionId = 'eamUnitTest',
   solutionName = 'eamUnitTest',
   agency = 'Agency:843b60d1-####-####-####-########c6d:null',
   agent = 'Agent:9325a7d2-####-####-####-########196:null',
   cluster = 'ClusterComputeResource:domain-c56:3b47f7fd-####-####-####-########33a',
   url = 'https://10.93.##.###:8080/vm/vm.ovf',
}

Environment

VMware vCenter Server 8.0.2
VMware vCenter Server 8.0

Cause

Any of the below mentioned cases can cause the issue:

The file server that hosts the OVF and/or VIB URLs of an EAM Agency uses an SSL certificate and:
There is an SSL certificate hostname mismatch, or
The SSL certificate is invalid or
The SSL certificate is self-signed, or
The SSL certificate is not trusted by the system. That is, the certificate is not signed by any of the root CA certificates of Photon S or VECS TRUSTED_ROOTS
The SSL certificate is provided via the EAM API (Agent.ConfigInfo.ovfSslTrust) and/or (Agent.ConfigInfo.vibSslTrust) or via the script /usr/lib/vmware-eam/bin/eam-utility.py and:
There is an SSL certificate hostname mismatch, or
The SSL certificate is invalid, or
The SSL certificate does not match the SSL certificate of the file server that hosts the OVF and/or VIB URLs of an EAM Agency.

Resolution

To resolve the issue please follow any one of the options mentioned below:

Option 1: Configure SSL trust via EAM API

Use EAM API properties Agent.ConfigInfo.ovfSslTrust and/or Agent.ConfigInfo.vibSslTrust to:

Configure a leaf SSL certificate that is to be trusted for a specific VIB or OVF URL.
Disable the SSL certificate verification when trying to access a specific OVF or VIB URL.

Option 2: Change the file server SSL certificate

Replace the file server SSL certificate with a valid one or such that is signed by any of Photon OS CAs or VECS TRUSTED_ROOTS CAs.

Option 3: Add a trusted root CA certificate to VECS

Add the root CA certificate signing the file server certificate to VECS TRUSTED_ROOTS.

Please refer Add a Trusted Root Certificate to the Certificate Store using the vSphere Client and vecs-cli Command Reference for more information.

Workaround:

Please follow one of the below mentioned options to configure the SSL trust via an EAM script.

Option 1: Configure a leaf SSL certificate that is to be trusted for a specific VIB or OVF URL

Login to VCSA through SSH using root.
Run the below command:

#/usr/lib/vmware-eam/bin/eam-utility.py install-cert <VIB/OVF URL>

Note:

The operation above can be reverted by running: eam-utility.py uninstall-cert <VIB/OVF URL>
The agency owner can also do the SSL trust configuration via the EAM API. In this case, it takes precedence over the configuration made via the script /usr/lib/vmware-eam/bin/eam-utility.py.

Option 2: Disable the SSL certificate verification for a specific VIB or OVF URL

Login to VCSA through SSH using root.
Run the below command:

#/usr/lib/vmware-eam/bin/eam-utility.py disable-trust <VIB/OVF URL>

Note:

The operation above can be reverted by running: eam-utility.py enable-trust <VIB/OVF URL>
The agency owner can also do the SSL trust configuration via the EAM API. In this case, it takes precedence over the configuration made via the script /usr/lib/vmware-eam/bin/eam-utility.py.