How to safely shutdown all nodes within a VCF Services Runtime Cluster

Products

• VCF Automation

Issue/Introduction

This article describes how to safely shut down a VCF Management Services deployment and power off the underlying virtual machines. The procedure applies to both the Fleet cluster and Instance clusters that host VCF management components.

The vcf_services_runtime_shutdown.sh script automates the full shutdown sequence via the API (port 5480) on the control plane node.

The script triggers a system-wide shutdown workflow that handles internal dependency ordering, prechecks (including snapshot detection), gracefully scales down workloads, and provisions flags to ensure automatic recovery upon boot.

Environment

• VCF Management Services Runtime 9.1.0.0
• VCF Automation 9.1.0.0

Cause

Use this procedure when you need to:

• Perform planned maintenance on the underlying vSphere infrastructure.
• Gracefully shut down the environment for a datacenter power event.
• Prepare the environment for a cold backup or VM snapshot.
• Decommission a VCF Management Services deployment.

Warning: This procedure stops all VCF management components running on the platform, including Log Management, Realtime Metrics, VCF Operations Lifecycle and Configuration Management, and others. Notify all dependent systems and end-users before proceeding.

Resolution

The vcf_services_runtime_shutdown.sh script can be run from any machine that has network access to port 5480 on at least one VCF Services Runtime cluster control plane node.

 

Preparation

1. Download the attached script, save it to a local file, and make it executable.
   1. chmod +x vcf_services_runtime_shutdown.sh
2. Ensure the following tools are installed
   1. curl: Communicates with the APIs on control plane node.
   2. jq: Parses JSON API responses.
   3. govc: Powers off VMs via vSphere.

 

Then identify a control plane node of the VCF services runtime as follows, and note down its IP address:

1. Open VCF Operations UI and navigate to Build > Lifecycle > Components tab
2. Click on the VCF Services Runtime link.
3. Scroll down the page to the Nodes section.
4. Identify a Control Plane node and take note of the IP address
5. Obtain the kubeconfig file from that node by doing following steps:
   1. ssh vmware-system-user@<node-ip>
   2. sudo -i
   3. Copy contents of /etc/kubernetes/admin.conf to a local file called "kubeconfig"

Dry Run (Recommended First Step)

Logs all planned actions without executing any shutdown or power-off operations:

./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> --dry-run  --kubeconfig <kubeconfig-file>

System Only (No VM Power-Off)

Shuts down the system but leaves VMs running. Useful when VMs need to remain accessible, or when vCenter credentials are not available:

./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> --skip-poweroff --kubeconfig <kubeconfig-file> 

Automated VM Power-Off

Shuts down the system and then powers off VMs automatically. Requires vCenter credentials:

export [email protected]
export VCENTER_PASSWORD=<vcenter-password>
./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> --kubeconfig <kubeconfig-file> 

vCenter URL is auto-discovered from the vsp component configuration. Set it manually to override:

export GOVC_URL=https://<vcenter-address>

Additional Information

Monitoring Progress

The script writes timestamped log output to stdout. Each phase is marked with a clear separator:

==========================================================
[2026-03-06T10:00:00Z] [STEP]  Shutting down VCF Services Runtime system
==========================================================
[2026-03-06T10:00:01Z] [INFO]  System shutdown task created: task-456
[2026-03-06T10:00:17Z] [INFO]  Task task-456 status: Succeeded
...

To monitor in real time and save a log file:

./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> --kubeconfig <kubeconfig-file> 2>&1 | tee /tmp/vcf_shutdown_$(date +%Y%m%d_%H%M%S).log