Official Procedure for Shutting Down VKS Infrastructure Stack
Article ID: 446899
Updated On:
Products
Issue/Introduction
Administrative procedure outlining the required sequence for gracefully shutting down the vSphere Kubernetes Service (VKS) infrastructure. This sequence ensures the Workload Control Plane, Supervisor Clusters, Workload Clusters, and underlying SDDC components (NSX, Avi, vSAN) are powered off without data corruption or workload state inconsistencies.
Cause
Planned administrative infrastructure shutdown, maintenance window, or data center power-off operations.
Resolution
Before initiating the power-off sequence, ensure the following:
- Backup: Confirm that vSphere Supervisor, NSX , VCF-Ops , vCenter, and NSX Manager have successful recent backups. All applications running on the workload domain on virtual machines and VKS Clusters should also have an application backup.
- Access: Ensure you have the root and admin credentials for all appliances (vCenter, NSX, Avi).
- Password & Certificate Validity - Ensure that passwords & certificates are not near expiry of this activity.
- DNS/NTP: Verify that external services like DNS and NTP remain active during the shutdown.
- Maintenance Mode: Ensure you have a "jump/bastion host" and local access to ESXi hosts using shell & dcui.
- Inventory: Use Powercli cmdlets or RvTools to export the inventory of the vCenter
1. Shutdown Applications on VKS Clusters The workload layer must be silenced first to prevent the Kubernetes control plane from attempting to restart pods during the infrastructure shutdown. Log in to the VKS clusters and gracefully shut down any running application services.
2. Cordon VKS Cluster Nodes Log in to individual VKS clusters and cordon the worker nodes to prevent new pod scheduling. kubectl cordon <node_name>
3. Stop Workload Control Plane Service on vCenter Log in to the vCenter Server via the vCenter Server Appliance Management Interface (VAMI) and stop the Workload Control Plane (WCP) Service. Alternatively, execute the following command from the vCenter shell: vmon-cli -k wcp
4. Pause VKS Clusters Log in to the Supervisor Control Plane Node and pause all VKS Clusters. Retrieve all clusters: kubectl get clusters Pause the clusters: kubectl patch cluster <cluster-name> -n <cluster-namespace> --type merge -p '{"spec":{"paused":true}}'
5. Shutdown Supervisor Control Plane Nodes Log in as root to the ESXi host managing the supervisor control plane nodes. Shut down the virtual machines directly from the ESXi Host UI or via the ESXi Shell: vim-cmd vmsvc/power.shutdown <vmid>
6. Shutdown VKS Cluster Nodes Power off the worker nodes first, followed by the control plane nodes for each VKS cluster using the vCenter UI.
7. Shutdown AVI Controllers Log in to the Avi Controller cluster. Shut down the follower nodes first. Shut down the leader node last using the vCenter UI.
8. Shutdown AVI Service Engines Shut down the AVI Service Engines directly from the vCenter UI.
9. NSX Networking Shut down the NSX Edge Transport Nodes (Edge VMs) providing North-South routing and Load Balancing services via the vCenter UI. Shut down the NSX Manager nodes using the vCenter UI one at a time.
10. vSAN Clusters and ESXi Hosts
Verify all virtual machines on the cluster are powered off.
Verify there are no resyncing objects on the vSAN Cluster.
Increase the object repair time to accommodate the duration of the outage using the steps detailed in Broadcom KB 327031.
Deactivate cluster member updates from vCenter by executing the following command on all ESXi hosts in the cluster:
esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdatesDeactivate vSphere HA on the cluster.
Prepare the vSAN cluster for shutdown by running the following command from any single ESXi host in the cluster:
python /usr/lib/vmware/vsan/bin/reboot_helper.py preparePlace hosts in Maintenance Mode:
esxcli system maintenanceMode set -e true -m noActionShutdown the ESXi hosts.
11. Shutdown vCenter Log in to the Management domain vCenter and shut down the vCenter Server appliance for the workload domain.
Additional Information
Feedback
