Error "Task Refresh database views failed" is encountered during the VMware Cloud Director database upgrade.

Products

• VMware Cloud Director

Issue/Introduction

• When upgrading VMware Cloud Director (VCD) from version 10.6.1.1 to 10.6.1.2, the database upgrade process fails during the schema modification phase due to an error related to database views.
  
  
• The upgrade utility terminates with the following error in the terminal: "Unable to upgrade the database: java.lang.IllegalStateException: Task Refresh database views failed."
  
  
• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log and upgrade-*.log, you can see the error as below:
  
  YYYY-MM-DD XX:XX:XX,XXX | WARN | pool-1-thread-1 | SerialAggregateTask | System_Public.sql: Task failed due to uncaught exception | org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:399) Caused by: java.net.SocketTimeoutException: Read timed out

Environment

VMware Cloud Director 10.6.1.x

Cause

This issue is caused by a communication timeout between the VCD upgrade utility and the PostgreSQL database backend. During the "Refresh database views" task, the database requires more time than the default 90-second timeout to process complex schema changes, resulting in a SocketTimeoutException and an I/O error at the driver level.

Resolution

To resolve this issue, increase the JDBC socket timeout configuration to allow the database task sufficient time to complete.

1. Ensure there is no significant network latency and no firewalls or load balancers between the VCD Cells and the Database server that could be dropping long-lived connections.

2. Create backups of the "global.properties" and "responses.properties" files located in /opt/vmware/vcloud-director/etc/.

3. On all cells, edit the "global.properties" and "responses.properties" files located in /opt/vmware/vcloud-director/etc/.

4. Locate the database.jdbcUrl line in both files and increase the socketTimeout value from 90 to 600.

   Example (Before): database.jdbcUrl = jdbc:postgresql://<IP_Address>:5432/vcloud?socketTimeout=90&ssl=true

   Example (After): database.jdbcUrl = jdbc:postgresql://<IP_Address>:5432/vcloud?socketTimeout=600&ssl=true

5. Restart the VMware Cloud Director service on all cells to apply the changes:
   
   service vmware-vcd restart 

6. Retry Upgrade: Re-run the VCD database upgrade utility:
   
   /opt/vmware/vcloud-director/bin/upgrade