VCF Operations Upgrade from 9.0 to 9.1 Stays Stuck With Inaccessible Replica Node

Products

• VCF Operations

Issue/Introduction

• When attempting to upgrade VMware Cloud Foundation (VCF) Operations from version 9.0 to version 9.1, the upgrade process stays stuck indefinitely at 9/10 step.
• Additionally, the replica node becomes completely inaccessible on the System Status page
• casa.log (/storage/vcops/log) on the VCF Operations primary node show:

2026-06-23T09:30:49.800Z INFO casa 10786 [ops@4413 threadId="1057153" threadName="pool-1046988-thread-1" requestId=" [com.vmware.vcops.casa.support.RetryRequestInterceptor. logException:125] - java.net.UnknownHostException: <VCF_operations_replica_node_fqdn>
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect (Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer (Unknown Source)

Output of the below command shows: 

ll /etc/resolv.conf 

Output:  /etc/resolv.conf
Expected Output: /etc/resolv.conf -> /run/systemd/resolve/resolv.conf 

Environment

VCF Operations 9.0.x

Cause

The issue is caused by a missing symbolic link for the DNS configuration file, which conflicts with security rules introduced in the VCF Operations 9.1.

Version 9.1 introduces a new AppArmor service configuration that explicitly grants read permissions to /run/systemd/resolve/resolv.conf. AppArmor expects /etc/resolv.conf to be a symbolic link pointing to that path.

If /etc/resolv.conf was created or modified as a standard regular file instead of a symlink, AppArmor denies the Customer Architecture and Solutions Engine (CASA) service read access to it. As a result, inter-node communication breaks down, throwing a java.net.UnknownHostException for the replica node in the logs.

Resolution

To resolve the upgrade lock and restore inter-node accessibility, manually recreate the expected symbolic link on both nodes by following these steps:

1. Revert to the upgrade snapshots taken on VCF Operations 9.0

2. Log into both the primary and replica nodes via SSH as root user.

3. Force-create the correct symbolic link pointing to the systemd-resolved file by running the following command:

   ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
   

4. Verify the link is correctly established using ll /etc/resolv.conf.

5. Restart the upgrade process.

Additional Information

This configuration issue will be natively resolved and automated in the upcoming VCF Operations 9.1.1 release.