Misconfigured no_proxy settings leads to failures in creating nfs service instance in VMware Tanzu Application Service (TAS) for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Prerequisites

• NFS Volume services enabled. Refer to Enable Volume Servicesfor more information.
• Configured Global Proxy settings for apps that are staging and/or running.
• Run the command cf running-environment-variable-group  to see the proxy settings configured with no_proxy not specifying *.cf.internal.

Symptoms 

cf create-service command instantly fails with the following error:

cf create-service nfs Existing si-nfs -c '{"share":"<IP-address>/share1"}'
Creating service instance si-nfs in org test / space test as admin... 

Service broker error: failed to store instance details: Error getting token. Your token may have expired and could not be refreshed. Please try logging in again. [Post https://uaa.service.cf.internal:8443/oauth/token: EOF] 
FAILED

Tracing the CF API calls made by the cf create-service command shows 

CF_TRACE=true cf create-service nfs Existing si-nfs -c '{"share":"<IP-address>/share1"}' > cf_trace.out

From the trace logs, you can see a 502 bad gateway logged:

REQUEST: [2020-06-29T18:41:24Z]
POST /v2/service_instances?accepts_incomplete=true HTTP/1.1
Host: #####
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/6.42.0+0cba12168.2019-01-10 (go1.11.4; amd64 linux)
{
  "name": "si-nfs",
  "parameters": {
    "share": "<IP-address>/share1"
  },
  "service_plan_guid": "6e87c98f-b546-4548-9022-1a0b3811c183",
  "space_guid": "b57cc6e2-1618-4a92-bc6b-2aa234293556"
}

RESPONSE: [2020-06-29T18:41:24Z]
HTTP/1.1 502 Bad Gateway
Content-Length: 503
Content-Type: application/json;charset=utf-8
Date: Mon, 29 Jun 2020 18:41:24 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: bb4bebb5-72e4-47ae-50ab-872aa097948f::f01241b5-eb6b-469c-aa79-6edefaaab7d2
{
  "code": 10001,
  "description": "Service broker error: failed to store instance details: Error getting token. Your token may have expired and could not be refreshed. Please try logging in again. [Post https://uaa.service.cf.internal:8443/oauth/token: EOF]",
  "error_code": "CF-ServiceBrokerBadResponse",
  "http": {
    "method": "PUT",
    "status": 500,
    "uri": "https://nfsbroker.sys.pivotal.io/v2/service_instances/e3cf45ee-4165-4cce-a685-26f6bc7eec7c?accepts_incomplete=true"
  }
}

Environment

Product Version: 2.7
OS: Linux

Resolution

This issue happens because the staging and running environment variable group was not flexible enough, specifically the no_proxy settings. This ends up blocking traffic intended for CF's internal domains (*.cf.internal), which includes uaa.service.cf.internal and credhub.service.cf.internal that nfsbroker uses to talk to respective services. 

The documentation, Set Environment Variables, mentions a note about including *.cf.internal in your no_proxy settings when configuring a staging or running environment variable group.

no_proxy is a comma-separated list of DNS suffixes or IP addresses that can be accessed without passing through the proxy. 
Pivotal recommends this value contains the domain .cf.internal, which is used by the platform to communicate with CredHub.

Follow the steps below to. resolve this problem:

1. Configure the correct no_proxy settings by modifying the cf set-running-environment-variable-group command and specifying *.cf.internal  as a DNS suffix for no_proxy

2. Restart the nfsbroker application that runs in system org and nfs space.