book
Article ID: 316821
calendar_today
Updated On:
Issue/Introduction
This article contains the information about Shutdown and Startup order for VMware PKS environment in case of any datacenter maintenance/Outages.
Resolution
In case of any datacenter maintenacne/outages, please follow the below order for shutting down / starting up the VMware PKS environment.
VMware PKS environment shutdown order:
- Disable BOSH Resurrection: If you have the Enable VM Resurrector Plugin checkbox selected in the BOSH Director tile > Director Config pane, you must turn BOSH resurrection off before restarting PKS, to prevent BOSH from recreating VMs. To do this, run the command bosh update-resurrection off
- Applications/Pods: Stop all the apps/pods running on the K8S Clusters.
- K8S Clusters:
- Stop all the K8S nodes in each cluster by following Shutdown the VMware PKS Cluster section in the VMware KB article, How to shutdown and startup a VMware PKS Cluster.
Note: all processes on K8s nodes will be stopped. And BOSH will mark them as stopped so when the VM is powered on, the process won't start automatically. - Shutdown all the K8S node VMs from the vCenter.
- Note: To shut down a virtual machine, right click the virtual machine in the VMware Host Client inventory, select Guest OS from the pop-up menu, and select Shut down. Follow the same to shutdown other virtual machines as well.
- PKS Control Plane:
- Stop the PKS API vm by using Bosh:
bosh -d pivotal-container-service-xxx stop - Shutdown the PKS API VM from the vCenter.
- Harbor:
- Stop the Harbor VM by using Bosh:
bosh -d harbor-container-registry-xxx stop - Shutdown the Harbor VM from the vCenter.
- Bosh: Shutdown the Bosh VM from the vCenter.
- Ops Manager: Shutdown the Ops Manager from the vCenter.
- NSX-T: Shutdown the below NSX-T related VMs from the vCenter.
- NSX-T Manager.
- NSX-T Controllers.
- NSX-T Edge nodes
- vCenter: Shutdown the vCenter, for more information on how to shutdown the vCenter, see Reboot or Shut Down the vCenter Server Appliance section in the VMware Documentation.
- ESXi Hosts: Put the ESXi’s in maintenance mode and then power them off (using iLO or other console methods)
VMware PKS environment Startup order:
- ESXi hosts: Power on all the ESXi hosts (using iLO or other console methods).
- vCenter: Connect to web interface of the ESXi hosting vCenter vm and power on the vm.
- NSX-T: Power on all the NSX-T related VMs from the vCenter.
- NSX-T Manager
- NSX-T Controllers
- NSX-T edge nodes
- Ops Manager: Power on the Ops Manager from the vCenter.
- Bosh:
- Power on the Bosh VM from the vCenter.
- If you are unable to connect to Bosh after powering on, see Unable to connect to Bosh client with an error "Requesting token via client credentials grant: UAA responded with non-successful status code '503' response 'FAILURE'" (67441)
- At this point in time, BOSH won’t try to resurrect any VMs because BOSH knows all the VMs under its control were BOSH stopped.
- PKS Control Plane VM:
- Power on the PKS API VM from the vCenter.
- If you are unable to connect to PKS VM after powering on, see PKS login fails with "Error: Unable to send a request to API: Head https://pks-api_FQDN:9021/v1/clusters: dial tcp pks-api-IP:9021: connect: connection refused" (65138).
- Harbor:
- Power on the Harbor VM from the vCenter.
- Verify the Harbor status using Bosh and start services if required.
- K8S Cluster nodes:
- Start the each K8S cluster by following the Startup the VMware PKS Cluster section in VMware KB article, How to shutdown and startup a VMware PKS Cluster.
- Applications/Pods : Start all the applications/pods as required.
- Re-enable BOSH Resurrection : Turn BOSH resurrection back on by running the command bosh update-resurrection on.