Shutdown and Startup order for VMware PKS environment

Products

• VMware Cloud PKS

Issue/Introduction

This article contains the information about Shutdown and Startup order for VMware PKS environment in case of any datacenter maintenance/Outages. 

Environment

VMware PKS 1.x

Resolution

In case of any datacenter maintenacne/outages, please follow the below order for shutting down / starting up the VMware PKS environment.

VMware PKS environment shutdown order:

1. Disable BOSH Resurrection: If you have the Enable VM Resurrector Plugin checkbox  selected in the BOSH Director tile > Director Config pane, you must turn BOSH resurrection off before restarting PKS, to prevent BOSH from recreating VMs. To do this, run the command bosh update-resurrection off
2. Applications/Pods: Stop all the apps/pods running on the K8S Clusters.
3. K8S Clusters:
   1. Stop all the K8S nodes in each cluster by following Shutdown the VMware PKS Cluster section in the VMware KB article, How to shutdown and startup a VMware PKS Cluster.
      Note: all processes on K8s nodes will be stopped. And BOSH will mark them as stopped so when the VM is powered on, the process won't start automatically.
   2. Shutdown all the K8S node VMs from the vCenter.
   3. Note: To shut down a virtual machine, right click the virtual machine in the VMware Host Client inventory, select Guest OS from the pop-up menu, and select Shut down. Follow the same to shutdown other virtual machines as well.
4. PKS Control Plane:
   1. Stop the PKS API vm by using Bosh:
      bosh -d pivotal-container-service-xxx stop
   2. Shutdown the PKS API VM from the vCenter.
5. Harbor:
   1. Stop the Harbor VM by using Bosh:
      bosh -d harbor-container-registry-xxx stop
   2. Shutdown the Harbor VM from the vCenter.
6. Bosh: Shutdown the Bosh VM from the vCenter.
7. Ops Manager: Shutdown the Ops Manager from the vCenter.
8. NSX-T: Shutdown the below NSX-T related VMs from the vCenter.
   1. NSX-T Manager.
   2. NSX-T Controllers.
   3. NSX-T Edge nodes
9. vCenter: Shutdown the vCenter, for more information on how to shutdown the vCenter, see Reboot or Shut Down the vCenter Server Appliance section in the VMware Documentation.
10. ESXi Hosts: Put the ESXi’s in maintenance mode and then power them off (using iLO or other console methods) 

VMware PKS environment Startup order:

1. ESXi hosts: Power on all the ESXi hosts (using iLO or other console methods).
2. vCenter: Connect to web interface of the ESXi hosting vCenter vm and power on the vm.
3. NSX-T: Power on all the NSX-T related VMs from the vCenter.
   1. NSX-T Manager
   2. NSX-T Controllers
   3. NSX-T edge nodes
4. Ops Manager: Power on the Ops Manager from the vCenter.
5. Bosh:
   1. Power on the Bosh VM from the vCenter.
   2. If you are unable to connect to Bosh after powering on, see Unable to connect to Bosh client with an error "Requesting token via client credentials grant: UAA responded with non-successful status code '503' response 'FAILURE'" (67441)
   3. At this point in time, BOSH won’t try to resurrect any VMs because BOSH knows all the VMs under its control were BOSH stopped.
6. PKS Control Plane VM:
   1. Power on the PKS API VM from the vCenter.
   2. If you are unable to connect to PKS VM after powering on, see PKS login fails with "Error: Unable to send a request to API: Head https://pks-api_FQDN:9021/v1/clusters: dial tcp pks-api-IP:9021: connect: connection refused" (65138).
7. Harbor:
   1. Power on the Harbor VM from the vCenter.
   2. Verify the Harbor status using Bosh and start services if required.
8. K8S Cluster nodes:
   1. Start the each K8S cluster by following the Startup the VMware PKS Cluster section in VMware KB article, How to shutdown and startup a VMware PKS Cluster.
9. Applications/Pods : Start all the applications/pods as required.
10. Re-enable BOSH Resurrection :  Turn BOSH resurrection back on by running the command bosh update-resurrection on.