Errands intermittently fail with EOF error when executing 'cf auth' on NetScaler

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

CF CLI fails with the following End of File (EOF) error when executing "cf auth"

cf auth system_verification [REDACTED]
API endpoint: https://api.system.domain.com
Authenticating...
Post https://login.system.domain.com/oauth/token: EOF
FAILED

Environment

Cause

When the CF CLI communicates with the Elastic Runtime UAA resource it will need to make two HTTP requests to UAA.

• GET /login
  • The expected response includes information about the login endpoints.
• POST /oauth/token
  • CF CLI expects the UAA to return an access token.

CF CLI reports an EOF error when sending the second request, "POST /oauth/token". This happens when NetScaler is the load balancer and configured to only allow a single request per connection.

When CF CLI sends the first GET request it is setting the HTTP header "Connection: close" which means the TCP connection will close after the response is received. NetScaler does not send back the same header in the response resulting in CF CLI attempting to reuse the same TCP connection for the post request. While sending the post request the NetScaler closes the TCP session and as a result, an EOF error is returned.

See the diagram to understand the scenario.

Resolution

The CF CLI fix is in release V6.36.1. The fix for Pivotal Application Service tile and other impacted services are listed below.

Tile and Product	Fix
Concourse	v3.11.0
PAS	2.1.4, 2.0.13, 1.12.22
MySQL	1.10.14, 2.3.0
Pivotal On Demand Service Broker SDK	0.21.1

Workaround for PAS 2.0

The new collocated errand feature in PAS 2.0 allows us to circumvent the name resolution of the login.system.domain.com URL.

1. When executing the errand review the BOSH task logs to identify which instance is executing the failing errand. Optionally you can run the errand manually and specify which instance you want the errand to run on with "--instance=" flag
2. Once the instance is identified, "BOSH SSH" to it and modify the "/etc/hosts" file to include an entry that points the login.system.domain.com to the IP address of one of the Gorouters.

   [IP OF GO ROUTER] login.SYSTEM.DOMAIN.COM

3. Run the errand manually by executing run command from inside the errand vm and confirm it completes successfully.  The errand script can be found in "/var/vcap/jobs/<errand>/bin/run".