TKGI / PKS login command fails with error: net/http Client timed out while awaiting headers

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Executing the TKGI (formerly PKS) login command fails with the following error:

Note: Make sure that you are seeing an IP address in the error. If an IP address is not in the error then the cause of timeout will be different and this article might not help to address that particular issue.

$ tkgi login -k -a <IP-Address> -u <username>
Password: *******

Error: Post https://10.###.###.13:8443/oauth/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Upon checking the logs of the TKGI API VM, specifically the pks-api monit jobs logs, you see the following error and its stack trace (filename: /var/vcap/sys/log/pks-api/pks-api.log):

2021-05-12 18:35:54.447 ERROR 27073 --- [nio-9021-exec-4] o.a.c.c.C.[Tomcat].[localhost]           : Exception Processing ErrorPage[errorCode
=0, location=/error] org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://10.###.###.13:8443/introspect": Hostname 10.###.###.13 not verified:
    certificate: sha256/redacted
    DN: <redacted>... CN=sandbox01-domain.com ...<redacted>
    subjectAltNames: []; nested exception is javax.net.ssl.SSLPeerUnverifiedException: Hostname 10.###.###.13 not verified:
    certificate: sha256/<redacted>
    DN: <redacted>... CN=sandbox01-domain.com ...<redacted>
    subjectAltNames: []
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web-5.1.18.RELEASE.jar!/:5.1.18.RELEASE]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:668) ~[spring-web-5.1.18.RELEASE.jar!/:5.1.18.RELEASE]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:577) ~[spring-web-5.1.18.RELEASE.jar!/:5.1.18.RELEASE]
        at org.springframework.security.oauth2.provider.token.RemoteTokenServices.postForMap(RemoteTokenServices.java:149) ~[spring-security-oauth2-2.3.6.RELEASE.jar!/:na]

The above error suggests that the certificates for the TKGI API is signed for sandbox01-domain.com hostname and does not have any alternative names (subjectAltNames: []). As you have specified an IP address while logging in to the API server, the hostname 10.###.###.13 could not be verified in the certificate. This leads to login attempts being considered a failure and ultimately, a timeout with the client timed out and awaiting headers.

Environment

Product Version: 1.9
OS: Linux

Resolution

To fix the issue, make sure to specify a hostname as opposed to an IP address at the location Ops Manager UI -> TKGI tile -> TKGI API -> API Hostname (FQDN). Upon saving the change, run an Apply Changes on the BOSH Director tile + TKGI tile (without the upgrade all instances errand) so the changes can be propagated to the API server VM. 

The example shown in the Issue section previously was fixed by changing the API Hostname (FQDN) in the tile from IP 10.###.###.13 to sandbox01-domain.com (the certificate was also generated for hostname sandbox01-domain.com).