cf push fails for foundations running Operations Manager 2.8.4 or Tanzu Application Service 2.8.4 with "Stats unavailable: Stats server temporarily unavailable."

Products

• Operations Manager

Issue/Introduction

Due to a known issue with BOSH DNS 1.18 which is found in Operations Manager 2.8.4, Diego instances are randomly not be able to resolve cf.internal addresses. This can lead to cf push failures and other issues in the foundation.

The release is not available for download any more. See for more informaiton: https://docs.pivotal.io/platform/2-8/release-notes/opsmanager-rn.html

The following errors are observed:

Waiting for app to start...
 
Unexpected Response
Response code: 503
CC code:       0
CC error code:
Request ID:    <request id>
Description:   {
  "description": "Stats unavailable: Stats server temporarily unavailable.",
  "error_code": "CF-StatsUnavailable",
  "code": 200002

To  temporarily workaround the problem, follow the steps below: 

1. From bosh cli do the following for all Diego instances and Diego cells:

bosh -d <cf-deployment>  ssh <instance> -c  "nslookup bbs.service.cf.internal"

Tip: If you run the command for the instance name without the ID, it will run it for all instances in the group. For example, the following command will return results from every Diego cell in the deployment:

bosh -d <cf-deployment>  ssh diego_cell -c  "nslookup bbs.service.cf.internal"

2. For all VMs that cannot resolve the cf.internal name, run the command below:

stdout | ** server can't find bbs.service.cf.internal: NXDOMAIN

------------------------------------------------------

bosh -d <cf-deployment> restart <instance_name/instance_id>

Environment

Product Version: 2.8

Resolution

The issue is resolved in Ops Manager 2.8.5 and above.