SDDC upgrade precheck fails due to HCX IX with error "vSphere SHA-1 validation failed"
search cancel

SDDC upgrade precheck fails due to HCX IX with error "vSphere SHA-1 validation failed"

book

Article ID: 377581

calendar_today

Updated On:

Products

VMware SDDC Manager VMware HCX

Issue/Introduction

  • Prechecks during preparation for an VCF-SDDC upgrade can give an error with the below message:
    ERROR vSphere SHA-1 validation failed
    High: Do not perform upgrade without addressing this issue.
    Check the /var/log/vmware/vcf/operationsmanager/assessment/pythonvalidations/###-###/artifacts/vsphere-sha1-validation-execution-error-###-###-###.txt file for more details. If that file contains error code 'rpc_s_connection_closed' then please retry the precheck as it could not connect to verify whether weak algorithms (e.g. SHA-1) are in use on the vCenter.



  • The log file in SDDC Manager highlighted in the above error will show more details. Example error below : 
    <timestamps> ERROR #################### Errors Found ####################
<timestamps> ERROR 
<timestamps> ERROR Support for certificates with weak signature algorithms has been removed in vSphere 8.0. Weak signature algorithm certificates must be replaced before upgrade. Refer to the vSphere release notes and VMware KB 89424 for more details. Correct the following 1 issues before proceeding with upgrade.
<timestamps> ERROR 
<timestamps> ERROR 1. Caught exception while validating host <HCX-MA-host-IP>: Access to perform the operation was denied.
<timestamps> ERROR 
<timestamps> ERROR ######################################################


  • Executing the script "vsphere8_upgrade_certificate_checks.py" provided in article Upgrading vCenter Server or ESXi 8.0 fails during precheck due to a weak certificate signature algorithm reports same error as above.

  • Run the below command from SSH to vCSA to confirm no SHA-1 certificate exists.
    for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Signature Algorithm"; done

  • Take SSH to vCSA and run below command to confirm HCX-MA (Mobility Agent) host is not using SHA-1 certificate. 
    openssl s_client -connect <HCX-MA-host-IP>:443


  • HCX-IX logs (/var/log/vmware/mobilityagent.log) show below : 
    <timestamps> info mobilityagent[02425] [Originator@6876 sub=Solo.Vmomi opID=###-### user=:VSPHERE.LOCAL\Administrator] Activation <<###-###, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 36042'>>, ha-certificate-manager, vim.host.CertificateManager.listCACertificates, <vim.version.v8_0_2_0, internal, 8.0.2.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x00007f6b5c009188]> : Invoke done [listCACertificates] on [vim.host.CertificateManager:ha-certificate-manager]
<timestamps> info mobilityagent[02425] [Originator@6876 sub=Solo.Vmomi opID=###-### user=:VSPHERE.LOCAL\Administrator] Throw vmodl.fault.SecurityError
2024-04-26T13:37:24.262Z info mobilityagent[02425] [Originator@6876 sub=Solo.Vmomi opID=###-### user=:VSPHERE.LOCAL\Administrator] Result:
--> (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = <unset>
-->    msg = ""
--> }


  • vCenter logs (/var/log/vmware/vpxd/vpxd.log) shows below :
    <timestamps> info vpxd[21498] [Originator@6876 sub=vpxLro opID=###] [VpxLRO] -- BEGIN task-320 -- ha-certificate-manager-85 -- vim.host.CertificateManager.listCACertificates -- ###-###-###(###-###-###)
<timestamps> info vpxd[21498] [Originator@6876 sub=vmomi.soapStub[131] opID=###] SOAP request returned HTTP failure; <<io_obj p:0x00007fedd5bc9478, h:77, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-85/sdk>, method: listCACertificates; code: 500(Internal Server Error); fault: (vmodl.fault.SecurityError) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fedd5bc9478, h:77, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-85/sdk>]: listCACertificates


Environment

  • VCF 5.0 or later ( vCenter 8.0 or later)
  • HCX 4.9.0 or earlier

Cause

HCX-IX Appliance adds the Mobility Agent service as a host object (VMware Mobility Platform) in the vCenter Server. It doesn't support some of the APIs like ListCACertificates() which is used by vCenter/SDDC Manager's to check certificates.
So when the script [vsphere8_upgrade_certificate_checks.py] runs against HCX-IX MA host, it fails to retrieve the certificates and throws error "Caught exception while validating host <HCX-MA-host-IP>: Access to perform the operation was denied."

Resolution

This issue is resolved in VMware HCX 4.9.1, available at Broadcom downloads.
HCX-IX needs to be upgraded even to 4.9.1 or later.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Additional Information

Powered by Wolken Software