BBR fails with EOF error when executing 'cf auth' on NetScaler

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
When running a BBR restore, an error similar to the one below is encountered:

pre-restore-lock failed: 1 error occurred:
error 1:
Error attempting to run pre-restore-lock for job backup-restore-notifications on backup_restore/<redacted>: +++ dirname /var/vcap/jobs/backup-restore-notifications/bin/bbr/pre-restore-lock
++ cd /var/vcap/jobs/backup-restore-notifications/bin/bbr/../..
++ pwd
+ JOB_PATH=/var/vcap/jobs/backup-restore-notifications
+ source /var/vcap/jobs/backup-restore-notifications/bin/common
++ set -ex
++ PATH=/var/vcap/packages/notifications-cf-cli/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
++ SCHEME=https
++ DOMAIN=X
++ API_ENDPOINT=https://api.system.domain.com
++ APP_NAME=notifications
++ APP_DOMAIN=X
++ set +x
++ ORG=system
++ SPACE=notifications-with-ui
++ BBR_SDK_PATH=/var/vcap/jobs/database-backup-restorer
++ CONFIG_JSON_PATH=/var/vcap/jobs/backup-restore-notifications/config/backup-restore-notifications-db-config.json
+ cf_auth_and_target
+ echo 'Authenticate and target...'
+ cf api https://api.system.domain.com --skip-ssl-validation
+ set +x
Post https://login.system.domain.com/oauth/token: EOF - exit code 1

Environment

Cause

When the CF CLI communicates with the Elastic Runtime UAA resource it will need to make two HTTP requests to UAA.

• GET /login
  • The expected response includes information about the login endpoints.
• POST /oauth/token
  • CF CLI expects the UAA to return an access token.

In this case, CF CLI reports an EOF error when sending the second request, "POST /oauth/token". This happens when NetScaler is the load balancer and configured to only allow a single request per connection.

When CF CLI sends the first GET request, it is setting the HTTP header "Connection: close" which means the TCP connection will close after the response is received.

NetScaler does not send back the same header in the response resulting in CF CLI attempting to reuse the same TCP connection for the post request. While sending the post request, the NetScaler closes the TCP session and as a result, an EOF error is returned.

This happens in CF CLI versions less than 6.36. The notifications errand ships with a lesser version CF CLI. Run the following command to confirm if the version used is affected:

backup_restore/id:~$ sudo find / -type f -name cf
/var/vcap/data/packages/cf-cli-6-linux/e07165814c382259307d2ae3a319641b59334c84/bin/cf
/var/vcap/data/packages/notifications-cf-cli/611aecc216d78780866f64b1c6b5ed0432ca2550/bin/cf

backup_restore/id:~$ /var/vcap/data/packages/notifications-cf-cli/id/bin/cf --version
cf version 6.31.0+b35df905d.2017-09-15

Note: See related KB for more details.

Resolution

To resolve this issue, follow the below steps: 

1. The CF CLI fix is in release V6.36.1. The workaround is to replace CF CLI with the latest version overwriting: /var/vcap/data/packages/notifications-cf-cli/id/bin/cf.

2. Upgrade to version 2.4.x.