How to safely shutdown all nodes within a VCF Services Runtime Cluster

Products

• VCF Automation

Issue/Introduction

This article describes how to safely shut down a VCF Management Services deployment and power off the underlying virtual machines. The procedure applies to both the Fleet cluster and Instance clusters that host VCF management components.

The vcf_services_runtime_shutdown.sh script automates the full shutdown sequence via the API (port 5480) on the control plane node.

The script triggers a system-wide shutdown workflow that handles internal dependency ordering, prechecks (including snapshot detection), gracefully scales down workloads, and provisions flags to ensure automatic recovery upon boot.

Environment

• VCF Management Services Runtime 9.1.0.x

Cause

Use this procedure when you need to:

• Perform planned maintenance on the underlying vSphere infrastructure.
• Gracefully shut down the environment for a datacenter power event.
• Prepare the environment for a cold backup or VM snapshot.
• Decommission a VCF Management Services deployment.

Warning: This procedure stops all VCF management components running on the platform, including Log Management, Realtime Metrics, VCF Operations Lifecycle and Configuration Management, and others. Notify all dependent systems and end-users before proceeding.

Resolution

Preparation

1. Upload the attached script to any Linux machine that have access to port 5480 on at least one VCF Services Runtime cluster control plane node.
2. Make it executable:

   1. chmod +x vcf_services_runtime_shutdown.sh

3. Obtain the breakglass password for the vmware-system-user account on the VCF Services Runtime cluster.
4. Ensure the following tools are installed
   1. curl: Communicates with the APIs on control plane node.
   2. jq: Parses JSON API responses.
   3. Optional: govc - automatically Powers off VMs via vSphere.

Identify a control plane node of the VCF services runtime:

1. Open VCF Operations UI and navigate to Build > Lifecycle > Components tab.
2. Click on the VCF Services Runtime link.
3. Scroll down the page to the Nodes section.
4. Identify a Control Plane node and take note of the IP address.

Dry Run (Recommended First Step)

Logs all planned actions without executing any shutdown or power-off operations:

./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> --dry-run

provide the vmware-system-user account when prompted.

System Only (No VM Power-Off)

Shuts down the system but leaves VMs running. Useful when VMs need to remain accessible, or if you do not have govc installed.

./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> 

provide the vmware-system-user account when prompted.

Note: The script will output the VMs that needs to be stopped from the vCenter server. Locate each VM -> right click -> Power -> Shut Down Guest OS, Confirm the Shut Down action.

Automated VM Power-Off (Requires govc and vCenter credentials)

Shuts down the system and then powers off VMs automatically.

export VCENTER_USERNAME=[e-mail]
export VCENTER_PASSWORD=<vcenter-password>
./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> --poweroff-vms

provide the vmware-system-user account when prompted.

Note: vCenter URL is auto-discovered from the VCF Services component configuration. If needed you could set it manually with:

export GOVC_URL=https://<vcenter-address>

Additional Information

Monitoring Progress

The script writes timestamped log output to stdout. Each phase is marked with a clear separator:

==========================================================
[2026-03-06T10:00:00Z] [STEP]  Shutting down VCF Services Runtime system
==========================================================
[2026-03-06T10:00:01Z] [INFO]  System shutdown task created: task-456
[2026-03-06T10:00:17Z] [INFO]  Task task-456 status: Succeeded
...

To monitor in real time and save a log file:

./vcf_services_runtime_shutdown.sh --node-ip <NODE_IP> 2>&1 | tee /tmp/vcf_shutdown_$(date +%Y%m%d_%H%M%S).log