vCenter Server Upgrade Fails Due to Stopped Content Library Service

Products

• VMware vCenter Server

Issue/Introduction

When attempting to upgrade VMware vCenter Server from version 7.0 Update 3i to a newer version, the upgrade process may fail with an error indicating that the Content Library Service has stopped. This issue prevents the successful completion of the vCenter Server upgrade.

Environment

• VMware vCenter Server
• Content Library Service stopped prior to upgrade attempt, or
• Content Library tables are not present or have invalid data.
  • See the Resolution section for steps for determining if the necessary Content Library tables are present and contain valid data.

Cause

The Content Library Service relies on specific database tables and configurations to function properly. If these tables are missing, have invalid values, or the service was stopped before the upgrade, it can cause the vCenter Server upgrade to fail.

Resolution

Verify whether the required database tables exist or contain invalid values:

1. Connect to the vCenter Server Appliance (VCSA) using SSH:
   a. Open an SSH client (e.g., PuTTY, Terminal).
   b. Enter the IP address or hostname of the VCSA.
   c. Provide the root username and password when prompted.

2. Access the PostgreSQL database:
   a. Connect to the PostgreSQL database using the following command:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

3. Verify the Content Library database tables:
   a. Run the following query to check the existence of the required tables:
      SELECT table_name FROM information_schema.tables WHERE table_schema='vc' AND table_name LIKE 'cl_%';
   b. Review the output to ensure that the necessary Content Library tables are present.

4. Check the contents of the cl_config table:
   a. Run the following query to view the contents of the cl_config table:
      SELECT * FROM cl_config;
   b. Verify that the database.schema.version value is set correctly.

5. Verify the existence and contents of the cl_subscription table:
   a. Run the following query to check if the cl_subscription table exists and contains records:
      SELECT * FROM cl_subscription;
   b. If records are present, share the output and do not proceed with further steps.
   c. If no records are present, drop the table using the following command:
      DROP TABLE cl_subscription;
   d. Recreate the table using the provided SQL statement (refer to the "Internal Notes" section in the previous knowledgebase article draft).

6. Check the contents of the cl_library table:
   a. Run the following query to view the contents of the cl_library table:
      SELECT * FROM cl_library;
   b. Review the output to ensure the table contains valid data.

7. Perform additional tests (optional):
   a. Run the following queries to investigate related tables:
      - SELECT * FROM cl_libraryitemhistory;
      - SELECT * FROM cl_libraryItemVersion;
      - SELECT * FROM cl_security_rule;
      - SELECT * FROM cl_security_policy_rule;
      - \d cl_library;
      - \d cl_subscription;
      - \d cl_libraryitemhistory;
      - \d cl_libraryItem;
      - \d cl_registry;
      - \d cl_registry_project_member;
   b. Review the output of each query to gather more information about the Content Library database.

8. Exit the PostgreSQL database:
   a. Type \q and press Enter to exit the PostgreSQL database.

9. Disconnect from the vCenter Server Appliance:
   a. Type exit and press Enter to close the SSH connection.

By following these steps, you can connect to the vCenter Server Appliance, access the PostgreSQL database, and perform the necessary tests to verify the presence and validity of the Content Library database tables.

Remember to replace the placeholders (e.g., IP address, hostname, passwords) with the actual values specific to your vCenter Server environment.

10. Open a case with VMware by Broadcom. Provide:
      a. The details of the commands you have run
      b. The exact version, including build number, of your vCenter Server Appliance

VMware may request further information, and will have to provide customized commands to repair the database, depending on the data you have provided.