Error: Something went wrong with the VxRail Manager server that responded 500 unexpectedly. Check and try again later.
search cancel

Error: Something went wrong with the VxRail Manager server that responded 500 unexpectedly. Check and try again later.

book

Article ID: 400679

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

vXRail plugin is not accessible indicating certificates downloaded from vCenter are invalid.

 

vsphere_client_virgo.log:

[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha256: #################################################################
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha1: ############################################
[[timestamp]] [ERROR] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Server certificate chain is not trusted and thumbprint doesn't match
[[timestamp]] [INFO ] -nio-127.0.0.1-5090-exec-204  org.apache.http.impl.execchain.RetryExec                          I/O exception (org.bouncycastle.tls.TlsFatalAlert) caught when processing request to {s}->https://VCENTER_IP:443: certificate_unknown(46)
[[timestamp]] [INFO ] -nio-127.0.0.1-5090-exec-204  org.apache.http.impl.execchain.RetryExec                          Retrying request to {s}->https://VCENTER_IP:443
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Trusted thumbprints: [##############################################, ##################################################################]
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha256: #################################################################
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha1: ############################################
[[timestamp]] [ERROR] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Server certificate chain is not trusted and thumbprint doesn't match
[[timestamp]] [INFO ] -nio-127.0.0.1-5090-exec-204  org.apache.http.impl.execchain.RetryExec                          I/O exception (org.bouncycastle.tls.TlsFatalAlert) caught when processing request to {s}->https://VCENTER_IP:443: certificate_unknown(46)
[[timestamp]] [INFO ] -nio-127.0.0.1-5090-exec-204  org.apache.http.impl.execchain.RetryExec                          Retrying request to {s}->https://VCENTER_IP:443
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Trusted thumbprints: [##############################################, ##################################################################]
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha256: #################################################################
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha1: ############################################
[[timestamp]] [ERROR] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Server certificate chain is not trusted and thumbprint doesn't match
[[timestamp]] [INFO ] -nio-127.0.0.1-5090-exec-204  org.apache.http.impl.execchain.RetryExec                          I/O exception (org.bouncycastle.tls.TlsFatalAlert) caught when processing request to {s}->https://VCENTER_IP:443: certificate_unknown(46)
[[timestamp]] [INFO ] -nio-127.0.0.1-5090-exec-204  org.apache.http.impl.execchain.RetryExec                          Retrying request to {s}->https://VCENTER_IP:443
[[timestamp]] [INFO ] agw-token-acq18              ######## ###### 200033 com.vmware.identity.token.impl.SamlTokenImpl                      SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
[[timestamp]] [INFO ] agw-token-acq18              ######## ###### 200033 com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl           Successfully acquired token for user: {Name: Administrator, Domain: VSPHERE.LOCAL}
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Trusted thumbprints: [##############################################, ##################################################################]
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha256: #################################################################
[[timestamp]] [WARN ] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      The server peer certificate thumbprint sha1: ############################################
[[timestamp]] [ERROR] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.ssl.ThumbprintTrustManager                      Server certificate chain is not trusted and thumbprint doesn't match
[[timestamp]] [ERROR] -nio-127.0.0.1-5090-exec-204  com.vmware.vxrail.mvc.ServiceProxyController                      Error occurred when calling VxRail Manager REST API org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at org.bouncycastle.jsse.provider.ProvSSLSocketWrap.checkServerTrusted(ProvSSLSocketWrap.java:131)
        at org.bouncycastle.jsse.provider.ProvTlsClient$1.notifyServerCertificate(ProvTlsClient.java:382)

 

 

 

 

Environment

vCenter 8.0 with vXRail plugin.

Cause

Wrong trusted root CRL on vCenter.

Resolution

Remove invalid trusted roots and regenerate valid ones.

Use the script crl-fix.sh attached to remove the extra entries in the TRUSTED_ROOT_CRLS store

  1. Login to the vCenter Server appliance node via ssh

  2. Capture the number of entries in the TRUSTED_ROOTS store
    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Number

  3. Capture the number of entries in the TRUSTED_ROOT_CRLS store
    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number

    Output sample:
    Number of entries in store :    1234

  4. Download the "crl-fix.sh" script attached to this KB and upload to the impacted VCSA.

  5. cd to /tmp folder

  6. Run chmod +x crl-fix.sh to make the file executable.

  7. Run ./crl-fix.sh
    If the error 'bash:  ./delete-subscription.sh: /bin/bash^M: bad interpreter: No such file or directory' is reported when the script is run, it is caused by DOS carriage returns that are added to the script when copying the script file from a Windows-based text editor.
    To resolve this problem, run the following command and then rerun the script:  sed -i -e 's/\r$//'  crl-fix.sh to fix the script, then run it again.

  8. Restart services on all vCenters in your SSO domain if vCenter is on ELM.
    service-control --stop --all
    service-control --start --all


Note: To remove expired certificates in TRUSTED_ROOTS, refer to KB 326288 (Verify and remove CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)) 

 

Attachments

crl-fix.sh get_app
Powered by Wolken Software