VIC Admin UI fails with "authentication failed due to incorrect credential(s)"
search cancel

VIC Admin UI fails with "authentication failed due to incorrect credential(s)"

book

Article ID: 340356

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Logging in to the VCH VIC admin UI https://vCenter-IP-FQDN:2378 fails.

  • Error observed:
    authentication failed due to incorrect credential(s)

Environment

VMware vSphere Integrated Containers 1.0.x

Cause

The vicadmin process cannot validate the vCenter Server SSL certificate. This can happen if the vCenter Server certificate is not properly formed or connecting through a proxy that has SSL offloading implemented.

Resolution

Check with the network team and security team responsible for the datacenter if any SSL offloading is in place for the datapath between the VCH endpoint VM and both the vCenter Server and Platform Service Controller.

If the proxy option was used in the vic-machine create options then the connection to vCenter Server can be tested for SSL offloading by running this command:

curl -I -x http://proxy-IP-FQDN:8080 https://vCenter-IP-FQDN -v


Look at the issuer in the SSL connection to determine if this is being signed by the proxy:


### lines omitted for brevity ###
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=########; C=##
* start date: <MMM DD xx:xx:xx Year timezone>
* expire date: <MMM DD xx:xx:xx Year timezone>
* issuer: CN=########; DC=#####; DC=#####; C=##; ST=#######; O=######; OU=#########

Compare the issuer to the output of a direct connection by running this command:

curl -k https://vCenter-IP-FQDN -v

 

If they are different then SSL offloading is in place.

### lines omitted for brevity ###
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=#######; C=##
* start date: <MMM DD xx:xx:xx Year timezone>
* expire date: <MMM DD xx:xx:xx Year timezone>
* issuer: CN=##; DC=XXXXXX; DC=#####; C=##; ST=XXXXX; O=########; OU=##############

If SSL offloading is not in use, check the vCenter SSL certificate for the correct Subject and subject Alternative Name by using openssl from the VCH shell.

  1. Enable the shell on the VCH endpoint VM by using vic-machine debug.
  2. Find the VCH ID by running this command:
vic-machine-operating_system ls --target <https://vCenter-FQDN-IP/"DataCenterName"> -compute-resource <"Name of Cluster"> --user <vcenter-username> --password <password> --thumbprint <vCenter-certificate-fingerprint>
  1. Use the VCH ID to enable the shell by running this command:
vic-machine-operating_system debug --target <https://vCenter-FQDN-IP/"DataCenterName"> -compute-resource <"Name of Cluster"> --id <VCH-ID> --user <vcenter-username> --password <password> --thumbprint <vCenter-certificate-fingerprint> --enable-ssh
  1.  The output will include the IP to connect a SSH client too. The default login is "root" and "password". This and other options can be changed see links below.
  1. Run this command to get the vCenter Server certificate information:
​ echo | openssl s_client -connect <vCenter-IP-FQDN>:443 2>/dev/null | openssl x509 -noout -text
  1. Check the Subject and Subject Alternative Name. These should have the vCenter Server hostname and DNS FQDN at the minimum. If not the vCenter Server certificate may need to be re-issued and VCH reconfigured or redeployed with new thumbprint.
This is an example of a vCenter Server certificate that is incorrect because the forward and revers lookup DNS records were not configured with the FQDN when the vCenter Server was deployed. This certificate example would mean that the cert needs to be re-issued to include FQDN or the VCH will need to be created with the --target option using IP instead of FQDN.

Note: Generally IPs are avoided in certificates, but was done here for example purposes. Contact the security team responsible for the datacenter for the certificate policy and/or refer to PCI compliance requirements the datacenter is required to meet.

# echo | openssl s_client -connect xxx.xxx.x.x:443 2>/dev/null | openssl x509 -noout -text


You see output similar to:

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
##:##:##:##:##:##:##
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=##, DC=######, DC=#####, C=##, O=########
Validity
Not Before: <MMM DD xx:xx:xx Year timezone>
Not After : <MMM DD xx:xx:xx Year timezone>
Subject: CN=xxx.xx.x.x, C=US
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
### lines omitted for brevity ###
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
IP Address:xxx.xx.xx.xx
X509v3 Subject Key Identifier:
##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
X509v3 Authority Key Identifier:
keyid:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
Signature Algorithm: sha256WithRSAEncryption
### lines omitted for brevity ###

Corrective Action once you have determined why the vicadmin cannot validate the vCenter Server.

  • If using SSL offloading through a proxy or other net security device.
  1. Request the sign certificate and CA chain from the security team responsible for the proxy.
  2. Deploy a VCH that includes the signing certificate and CA chain in the vic-machine create option "--tls-ca".
  • If vCenter Server has the correct information in the certificate Subject and Subject Alternative Name.

a) Check the vicadmin.log file to find if the VCH admin UI is connecting by IP or FQDN. This example shows connections are being validated with FQDN:

[YYYY-MM-DDTHH:MM:SS] DEBUG [BEGIN] [main.(*server).loginPage:268]
[YYYY-MM-DDTHH:MM:SS] WARN Unable to connect: Failed to connect to vCenter-IP-FQDN: Post https://vCenter-IP-FQDN/sdk: Service Unavailable
[YYYY-MM-DDTHH:MM:SS] WARN xxx.xx.x.x:59510 failed to authenticate

The vicadmin.log file is located at /var/log/vic/ on the VCH endpoint VM. The log can be accessed in the following ways:
      1. Enter https://vCenter-IP-FQDN:2378/logs/vicadmin.log address in the browser.
      2. Enable the shell using vic-machine debug.
         
        ​​For more information see:
        • VCH Debug Options
        • Authorize SSH Access to the VCH Endpoint VM
      3. Once enabled connect a SSH session and run the more /var/log/vic/vicadmin.log command.

b) Redeploy the VCH setting the --target option to either IP or FQDN depending on what the vCenter Server certificate has for Subject and Subject Alternative Name.

c) Alternate option. Re-issue the vCenter Server certificate to include the IP or FQDN based on what the VCH admin UI is using to validate the vCenter connection based on the vicadmin.log file.

d) After the vCenter Server certificates are replaced, get the new vCenter Server fingerprint using openssl from a shell session on the VCH. 

echo | openssl s_client -connect <vCenter-IP-FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint

e) Update the fingerprint to the new vCenter Server fingerprint in the VCH vmx configuration file on the ESXi host datastore the VCH VM is on for the value:

 guestinfo.vice./connect/target_thumbprint = <fingerprint>.

f) Shutdown the VCH vApp completely then power on from vApp to apply changes.  



Powered by Wolken Software