Foundation is not reachable, AppsManager is not available and unable to SSH into the Opsmanager too

Products

• VMware Tanzu Application Service

Issue/Introduction

• TAS and TKGI is unreachable
• After a network change, users are unable to SSH into Opsman VM and the Foundation is no longer reachable.
• After the network change has been reverted, and Opsman SSH connectivity is restored, TAS VM's report 'failing' and 'starting' states. Apps Manager is still inaccessible. 
• When reviewing the CF instance you see the 'clock_global' and 'cloud_controller_workers' are stuck in 'failing' state. The 'cloud_controller' is in 'starting' state.
• When you SSH into one of the 'clock_global' VM's and run sudo monit summary, you see the 'cloud_controller_clock' and 'cc_deployment_updater' services are stuck in 'Does not exist' state.
• When checking the cloud_controller_clock.log from the 'clock_global' VM, you see errors indicating:
  
  Mysql2::Error::ConnectionError: Lost connection to server at 'handshake: reading initial communication packet', system error: 11 (Mysql2::Error::ConnectionError)


• When checking the MySQL VMs from bosh in the CF instance, you see that all are in a running state.
• From an SSH into the 'mysql_monitor' VM, run the following command to check the MySQL diagnostics:
  
  # mysql-diag


• You will see a [CRITICAL] warning indicating: "You must bootstrap the cluster."
  
  

Environment

This problem was observed on TAS 3.3, but is not isolated to this version.

Cause

This problem occurs because the network changes applied break quorum for the internal MySQL cluster backing TAS components. Once the network is restored, the MySQL servers are unable to re-establish quorum without manual intervention. See the Recovering from MySQL Downtime section of the Managing Internal MySQL under TAS for VMs documentation for more details.

Resolution

• Run the bootstrap errand manually to bootstrap the MySQL cluster and return quorum:
  
  # bosh -d <CF-INSTANCE> run-errand bootstrap
  
  

CAUTION: Please ensure you run this bootstrap errand against the MySQL instance running in the <CF-DEPLOYMENT>. Ensure you are not running against a MySQL deployment created by the Tanzu for MySQL Tile.