VCF Operations upgrade from 9.0 to 9.1 stuck at 10/14 - "Applied Product Update stage"

Products

• VCF Operations

Issue/Introduction

• VCF Operations upgrade from, version 9.0 to 9.1 stays stuck at the below stage for all Analytics nodes.

10/14 - Applied Product Update stage

• The cluster status on the System Status page shows 'Going Online' 
• In the casa.log (/storage/vcops/log/casa-webapp) show:

2026-07-13T12:54:05.580Z INFO casa #### [####@#### threadId="####" threadName="ajp-nio-####-exec-####" requestId="####"] [com.vmware.vcops.casa.support.subprocess.GeneralCommand.execute:257] - Command '/usr/bin/sudo -n /usr/bin/bash /usr/lib/vmware-vcopssuite/utilities/security-wrapper/safe-ntpdate.sh -u ####' threw exception: CommandLineExitException: key=general.failure; args=126,/usr/lib/vmware-vcopssuite/utilities/security-wrapper/safe-ntpdate.sh: line 14: /usr/bin/ntpdate: Permission denied; cause=

2026-07-13T12:54:05.581Z WARN casa #### [####@#### threadId="####" threadName="ajp-nio-####-exec-####" requestId="####"] [com.vmware.vcops.casa.exception.CasaControllerExceptionHandler.runtimeExceptionHandler:276] - general runtime exception com.vmware.vcops.casa.exception.CasaException: Failure while forcing NTP sync; see operating system NTP log

Environment

• VCF Operations 9.0 
• VCF Operations 9.1 

Cause

This issue occurs due to an NTP time drift in the cluster, which prevents the upgrade script from proceeding.

Resolution

To resolve this issue, ensure the NTP configuration is correct and synchronized across all nodes:

1. Log in to the VCF Operations Admin UI page.
2. Verify the NTP server details (located at the top right side) to ensure they are correct.
3. Validate that the time is synchronized and identical across all cluster nodes. 
4. Update the NTP source if the current one is inaccurate.

Once the NTP drift is corrected, the upgrade script resumes automatically.