Upgrade Pre-check states "Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!"

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

During the Pre-check phase of vCenter Server upgrade or patching, any of following errors are observed in the UI :

Pre-check error while upgrading vCenter Server to 8.0 U2:
"Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!"
"Verify following URLs and their respective statuses and follow KB 93526.
<LIST_OF_URLs>"

Pre-check error while upgrading vCenter Server to 8.0 U2a:
"Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!"
"Please refer to https://kb.vmware.com/s/article/93526 to trust the URLs: <LIST_OF_URLs>."

Pre-check error while patching vCenter Server to 8.0 U2a:

Upgrade scenario:

In the /storage/log/vmware/upgrade/CollectRequirements_com.vmware.eam_<DATE>.log file you see:

2023-10-16T10:07:24.375Z INFO eam.lib.eam-upgrade-prechecks Verifying ('IP/FQDN', 8080). 2023-10-16T10:07:24.906Z WARNING eam.lib.eam-upgrade-prechecks The ('IP/FQDN', 8080) is not trusted. Traceback (most recent call last):   File "/tmp/vmware-upgrade-temp-dir9Hsu3kuBng/tmpDa2KRuY8Zc/payload/component-scripts/eam/lib/trust-verifier.py", line 287, in _verifyServer     do_handshake_on_connect=True   File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket     _context=self)   File "/usr/lib/python3.5/ssl.py", line 760, in __init__     self.do_handshake()   File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake     self._sslobj.do_handshake()   File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake     self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719) ... 2023-10-16T10:07:25.477Z INFO extensions The component script returned '{'srcPorts': [], 'installArguments': {}, 'coreRequirement': {'importEstimationTime': 0, 'requiredSrcDiskSpace': 0.0, 'requiredDstDiskSpace': {}, 'exportEstimationTime': 0}, 'dstPortSpecs': [], 'userOptionSpecs': [], 'extraArguments': {}, 'requirementMismatchSpecs': [{'problemId': None, 'resolution': Please refer to https://kb.vmware.com/s/article/93526 to trust the URLs: https://<IP/FQDN>:8080/vm/vm.ovf., 'severity': 'ERROR', 'text': Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!, 'description': None}]}'

Patching Scenario:

In the /storage/log/vmware/applmgmt/PatchRunner.log file:

2023-10-13T20:15:22.385Z eam:CollectRequirements INFO eam.lib.eam-upgrade-prechecks Verifying ('IP/FQDN', 8080). 2023-10-13T20:15:22.872Z eam:CollectRequirements WARNING eam.lib.eam-upgrade-prechecks The ('IP/FQDN', 8080) is not trusted. Traceback (most recent call last):   File "/storage/core/software-update/updates/8.0.2.00100/scripts/patches/payload/components-script/eam/lib/trust-verifier.py", line 284, in _verifyServer     context.wrap_socket(   File "/usr/lib/python3.10/ssl.py", line 512, in wrap_socket     return self.sslsocket_class._create(   File "/usr/lib/python3.10/ssl.py", line 1070, in _create     self.do_handshake()   File "/usr/lib/python3.10/ssl.py", line 1341, in do_handshake     self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:997)    2023-10-13T20:15:45.482Z INFO vmware_b2b.patching.phases.discoverer Discovery completed. Result: [  ...             "name": "eam",         "patchScript": "/storage/core/software-update/updates/8.0.2.00100/scripts/patches/payload/components-script/eam",         "requirementsResult": {             "mismatches": [                 {                     "description": null,                     "problemId": null,                     "relatedUserDataId": null,                     "resolution": {                         "args": [                             "https://<IP/FQDN>:8080/vm/vm.ovf"                         ],                         "id": "eam.action.ensure.url.trusted",                         "localized": "Please refer to https://kb.vmware.com/s/article/93526 to trust the URLs: https://<IP/FQDN>:8080/vm/vm.ovf.",                         "translatable": "Please refer to https://kb.vmware.com/s/article/93526 to trust the URLs: %(0)s."                     },                     "severity": "error",                     "text": {                         "id": "eam.url.not.trusted",                         "localized": "Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!",                         "translatable": "Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!"                     }             }         }

Cause

In order to harden security, SSL trust pre-checks for the VIB/OVF URLs configured with ESX Agent Manager (EAM) are executed as one of the first steps during the VC upgrade.

The file server that hosts the OVF and/or VIB URLs of an EAM Agency uses an SSL certificate and any of the following is true:
- There is an SSL certificate hostname mismatch, or
- The SSL certificate is invalid, or
- The SSL certificate is not trusted by the system. This can happen if the certificate verification is enabled on the system and any of the following is true:
  - The certificate is not signed by any of the root CA certificates of Photon OS or VECS TRUSTED_ROOTS.
  - The certificate is not configured explicitly to be trusted by the system using the attached script.

Resolution

Any of the following options can be used as a remediation of the problem. After execution of any of the options, the VC upgrade needs to be started again.

Option 1 (recommended): Configure a leaf SSL certificate that is to be trusted for a specific VIB or OVF URL

Login to VC through SSH using root.
Download eam-utility.py script from this KB on the VC file system (script_location).
Run the below command (where the '#' represents the command line and does not need to be entered):

# python <script_location>/eam-utility.py install-cert <VIB/OVF/ZIP URL>

Retry the vCenter Upgrade.

NOTE:

The operation above can be reverted by running: eam-utility.py uninstall-cert <VIB/OVF/ZIP URL>
The full URL path including the VIB/OVF/ZIP is required.
While upgrading/updating vCenter Server to 8.0U2a or later, the URLs in GUI is upended with . (period). This has to be removed when running command.
ex) In the picture of Symptoms section Pre-check error while upgrading vCenter Server to 8.0 U2a: GUI displays
https://__MASK__:8080/vm/vm.ovf.
The last . (period) has to be removed and the correct URL is
https://__MASK__:8080/vm/vm.ovf
All URLs in the pre-check error are required to be added regardless if the SSL certificate hash is the same.

Option 2: Disable the SSL certificate verification for a specific VIB or OVF URL

Note: Using this option makes VIB and OVF URL download insecure.

Login to VC through SSH using root.
Download eam-utility.py script from this KB on the VC file system (script_location).
Run the below command (where the '#' represents the command line and does not need to be entered):

# python <script_location>/eam-utility.py disable-trust <VIB/OVF/ZIP URL>

Retry the vCenter Upgrade.

NOTE:

The operation above can be reverted by running: eam-utility.py enable-trust <VIB/OVF/ZIP URL>
The full URL path including the VIB/OVF/ZIP is required.
While upgrading/updating vCenter Server to 8.0U2a or later, the URLs in GUI is upended with . (period). This has to be removed when running command.
ex) In the picture of Symptoms section Pre-check error while upgrading vCenter Server to 8.0 U2a: GUI displays
https://__MASK__:8080/vm/vm.ovf.
The last . (period) has to be removed and the correct URL is
https://__MASK__:8080/vm/vm.ovf
All URLs in the pre-check error are required to be added regardless if the SSL certificate hash is the same.

Option 3: Change the file server SSL certificate

If the SSL certificate is invalid:

Replace the SSL certificate with a valid one.
Retry the vCenter Upgrade.

Option 4: Add a trusted root CA certificate to VECS

If the SSL certificate is issued by a CA,

Add the root CA certificate to VECS TRUSTED_ROOTS.
- Please refer to: Add a Trusted Root Certificate to the Certificate Store Using the vSphere Client and vecs-cli .
Retry the vCenter Upgrade.

Attachments

eam-utility get_app